Probabilities and Statistics refresher

Star

By Afshine Amidi and Shervine Amidi

Introduction to Probability and Combinatorics

Sample space The set of all possible outcomes of an experiment is known as the sample space of the experiment and is denoted by $S$.

Event Any subset $E$ of the sample space is known as an event. That is, an event is a set consisting of possible outcomes of the experiment. If the outcome of the experiment is contained in $E$, then we say that $E$ has occurred.

Axioms of probability For each event $E$, we denote $P(E)$ as the probability of event $E$ occurring.

Axiom 1 ― Every probability is between 0 and 1 included, i.e:

Axiom 2 ― The probability that at least one of the elementary events in the entire sample space will occur is 1, i.e:

Axiom 3 ― For any sequence of mutually exclusive events $E_1, ..., E_n$, we have:

Permutation A permutation is an arrangement of $r$ objects from a pool of $n$ objects, in a given order. The number of such arrangements is given by $P(n, r)$, defined as:

Combination A combination is an arrangement of $r$ objects from a pool of $n$ objects, where the order does not matter. The number of such arrangements is given by $C(n, r)$, defined as:

Remark: we note that for $0\leqslant r\leqslant n$, we have $P(n,r)\geqslant C(n,r)$.

Conditional Probability

Bayes' rule For events $A$ and $B$ such that $P(B)>0$, we have:

Partition Let $\{A_i, i\in[\![1,n]\!]\}$ be such that for all $i$, $A_i\neq\varnothing$. We say that $\{A_i\}$ is a partition if we have:

Remark: for any event $B$ in the sample space, we have $\displaystyle P(B)=\sum_{i=1}^nP(B|A_i)P(A_i)$.

Extended form of Bayes' rule Let $\{A_i, i\in[\![1,n]\!]\}$ be a partition of the sample space. We have:

Independence Two events $A$ and $B$ are independent if and only if we have:

Random Variables

Definitions

Random variable A random variable, often noted $X$, is a function that maps every element in a sample space to a real line.

Cumulative distribution function (CDF) The cumulative distribution function $F$, which is monotonically non-decreasing and is such that $\underset{x\rightarrow-\infty}{\textrm{lim}}F(x)=0$ and $\underset{x\rightarrow+\infty}{\textrm{lim}}F(x)=1$, is defined as:

Remark: we have $P(a < X\leqslant B)=F(b)-F(a)$.

Probability density function (PDF) The probability density function $f$ is the probability that $X$ takes on values between two adjacent realizations of the random variable.

Relationships involving the PDF and CDF Here are the important properties to know in the discrete (D) and the continuous (C) cases.

Expectation and Moments of the Distribution Here are the expressions of the expected value $E[X]$, generalized expected value $E[g(X)]$, $k^{th}$ moment $E[X^k]$ and characteristic function $\psi(\omega)$ for the discrete and continuous cases:

Variance The variance of a random variable, often noted Var$(X)$ or $\sigma^2$, is a measure of the spread of its distribution function. It is determined as follows:

Standard deviation The standard deviation of a random variable, often noted $\sigma$, is a measure of the spread of its distribution function which is compatible with the units of the actual random variable. It is determined as follows:

Transformation of random variables Let the variables $X$ and $Y$ be linked by some function. By noting $f_X$ and $f_Y$ the distribution function of $X$ and $Y$ respectively, we have:

Leibniz integral rule Let $g$ be a function of $x$ and potentially $c$, and $a, b$ boundaries that may depend on $c$. We have:

Probability Distributions

Chebyshev's inequality Let $X$ be a random variable with expected value $\mu$. For $k, \sigma>0$, we have the following inequality:

Main distributions Here are the main distributions to have in mind:

Type	Distribution	PDF	$\psi(\omega)$	$E[X]$	$\textrm{Var}(X)$	Illustration
(D)	$X\sim\mathcal{B}(n, p)$	$\displaystyle \displaystyle\binom{n}{x} p^xq^{n-x}$	$(pe^{i\omega}+q)^n$	$np$	$npq$
(D)	$X\sim\textrm{Po}(\mu)$	$\displaystyle \frac{\mu^x}{x!}e^{-\mu}$	$e^{\mu(e^{i\omega}-1)}$	$\mu$	$\mu$
(C)	$X\sim\mathcal{U}(a, b)$	$\displaystyle \frac{1}{b-a}$	$\displaystyle\frac{e^{i\omega b}-e^{i\omega a}}{(b-a)i\omega}$	$\displaystyle\frac{a+b}{2}$	$\displaystyle\frac{(b-a)^2}{12}$
(C)	$X\sim\mathcal{N}(\mu, \sigma)$	$\displaystyle \frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{1}{2}\left(\frac{x-\mu}{\sigma}\right)^2}$	$e^{i\omega\mu-\frac{1}{2}\omega^2\sigma^2}$	$\mu$	$\sigma^2$
(C)	$X\sim\textrm{Exp}(\lambda)$	$\displaystyle \lambda e^{-\lambda x}$	$\displaystyle\frac{1}{1-\frac{i\omega}{\lambda}}$	$\displaystyle\frac{1}{\lambda}$	$\displaystyle\frac{1}{\lambda^2}$

Jointly Distributed Random Variables

Marginal density and cumulative distribution From the joint density probability function $f_{XY}$ , we have

Conditional density The conditional density of $X$ with respect to $Y$, often noted $f_{X|Y}$, is defined as follows:

Independence Two random variables $X$ and $Y$ are said to be independent if we have:

Covariance We define the covariance of two random variables $X$ and $Y$, that we note $\sigma_{XY}^2$ or more commonly $\textrm{Cov}(X,Y)$, as follows:

Correlation By noting $\sigma_X, \sigma_Y$ the standard deviations of $X$ and $Y$, we define the correlation between the random variables $X$ and $Y$, noted $\rho_{XY}$, as follows:

Remark 1: we note that for any random variables $X, Y$, we have $\rho_{XY}\in[-1,1]$.

Remark 2: If X and Y are independent, then $\rho_{XY} = 0$.

Parameter estimation

Definitions

Random sample A random sample is a collection of $n$ random variables $X_1, ..., X_n$ that are independent and identically distributed with $X$.

Estimator An estimator is a function of the data that is used to infer the value of an unknown parameter in a statistical model.

Bias The bias of an estimator $\hat{\theta}$ is defined as being the difference between the expected value of the distribution of $\hat{\theta}$ and the true value, i.e.:

Remark: an estimator is said to be unbiased when we have $E[\hat{\theta}]=\theta$.

Estimating the mean

Sample mean The sample mean of a random sample is used to estimate the true mean $\mu$ of a distribution, is often noted $\overline{X}$ and is defined as follows:

Remark: the sample mean is unbiased, i.e $E[\overline{X}]=\mu$.

Central Limit Theorem Let us have a random sample $X_1, ..., X_n$ following a given distribution with mean $\mu$ and variance $\sigma^2$, then we have:

Estimating the variance

Sample variance The sample variance of a random sample is used to estimate the true variance $\sigma^2$ of a distribution, is often noted $s^2$ or $\hat{\sigma}^2$ and is defined as follows:

Remark: the sample variance is unbiased, i.e $E[s^2]=\sigma^2$.

Chi-Squared relation with sample variance Let $s^2$ be the sample variance of a random sample. We have: