Normal Distribution

The Normal distribution (or gaussian distribution) is defined between $-\mathrm{\infty }$ and $\mathrm{\infty }$. It is parametrized by mean $\mu$ and variance $\sigma$, $X\sim \mathcal{N}(\mu ,{\sigma }^2)$ $$\begin{array}{r}{f}_X(x)=\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{(x-\mu {)}^2}{2{\sigma }^2}}\end{array}$$ As already described, $$\begin{array}{rl}E[X]& =\mu \\ Var(X)& ={\sigma }^2\end{array}$$

Standard Normal Distribution

A Standard Normal is defined as a normal distribution with $\mu =0$ and ${\sigma }^2=1$ Any normal distribution can be converted to a standard normal as $X=\frac{X-\mu }{\sigma }$ If $Y=aX+b$, then $Y\sim \mathcal{N}(a\mu +b,a^2{\sigma }^2)$.

For a standard normal variable $Z$, it is standard to denote $$\begin{array}{r}P(Z\le z)=\mathrm{\Phi }(z)P(Z=z)=\varphi (z)=\mathcal{N}(0,1)\end{array}$$

Further, for a given $\alpha \in (0,1)$, define $z_{\alpha }$ by $$\begin{array}{r}P(Z>z_{\alpha })=\alpha =1-\mathrm{\Phi }(z_{\alpha })\end{array}$$ Some standard values of $\alpha$ can be useful

• $z_{0.01}=1.2816$

• $z_{0.05}=1.645$

• $z_{0.025}=1.96$

• $z_{0.01}=2.33$

The 68–95–99.7 rule is also useful which states that

• Probability of $X$ lying between 1 standard deviation on either side of mean is 68%
• Probability of $X$ lying between 2 standard deviation on either side of mean is 95%
• Probability of $X$ lying between 3 standard deviation on either side of mean is 99.7%

It is often easier to derive all formulae here by first considering the standard normal and defining it completely. Then any normal distribution with mean $\mu$ and variance ${\sigma }^2$ can be obtained using the transformation $Y=\sigma Z+\mu$. All calculations for mean, variance and mgf readily follow.

Moment Generating Function

$$\begin{array}{r}E[e^{tX}]={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{e}^{tx}\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{(x-\mu {)}^2}{2{\sigma }^2}}\end{array}$$ We will rearrange the terms to form a perfect square in the exponent part with a changed mean. $$\begin{array}{rl}E[e^{tX}]& ={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{e}^{tx}\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{(x-\mu {)}^2}{2{\sigma }^2}}\\ & ={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp (-\frac{(x-(\mu +{\sigma }^{2}t){)}^2-(\mu +{\sigma }^{2}t{)}^2+{\mu }^2}{2{\sigma }^2})\\ & ={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp (-\frac{(x-(\mu +{\sigma }^{2}t{)}^2}{2{\sigma }^2})\exp (\mu t+\frac{{\sigma }^2{t}^2}{2})\\ E[e^{tX}]& =\exp (\mu t+\frac{{\sigma }^2{t}^2}{2})\end{array}$$ Since the total integral of a normal distribution is 1 (the total probability).

Sum of Normal Distributions

With the help of moment generating functions, this calculation becomes easier. Let $X_1,\dots X_n$ be $n$ independent normal distributions with $X_i\sim \mathcal{N}({\mu }_i,{\sigma }_i^2)$. $$\begin{array}{rl}E[e^{t(X_1+X_2+\cdots +X_n)}]& =\prod _{i=1}^{n}E[e^{t{X}_i}]=\prod _{i=1}^n{e}^{{\mu }_{i}t+\frac{{\sigma }_i^2{t}^2}{2}}\\ & =\exp (\sum _{i=1}^n{\mu }_{i}t+\sum _{i=1}^n{\sigma }^2\frac{t^2}{2})\\ ⟹X_1+X_2+\cdots +X_n& \sim \mathcal{N}({\mu }_1+\cdots +{\mu }_2,{\sigma }_1^2+\cdots +{\sigma }_2^2)\end{array}$$

Multivariate Normal Distribution

Multivariate normal distribution is an extension of a normal distribution into multiple dimensions $$\begin{array}{r}{f}_{\mathbf{X}}(\mathit{x})=\frac{1}{\sqrt{(2\pi {)}^d\left|\mathrm{\Sigma }\right|}}exp(-\frac{1}{2}(\mathbf{x}-\mathit{\mu }){\mathrm{\Sigma }}^{-1}(\mathbf{x}-\mathit{\mu }{)}^T)\end{array}$$ for $d$ dimensional vector $\mathbf{x}$ with mean vector $\mathit{\mu }$ and covriance matrix $\mathrm{\Sigma }$.

This is easy to derive if we start from a standard normal first.

Consider $n$ independent identically distributed standard normals $Z_i\sim N(0,1)$. The joint distribution is $$\begin{array}{rl}{f}_{\mathbf{Z}}(\mathbf{z})& =\prod _{i=1}^n{f}_{Z_i}(z_i)={\left(\frac{1}{\sqrt{2\pi }}\right)}^n\exp (-\frac{1}{2}\sum _{i=1}^n{z}_i^2)\\ & ={\left(\frac{1}{2\pi }\right)}^{\frac{n}{2}}\exp (-\frac{1}{2}{\mathbf{z}}^T\mathbf{z})\end{array}$$

$E[\mathbf{Z}]=0=E[\mathbf{Z}{]}^T$ (element wise). $$\begin{array}{rl}Cov(\mathbf{Z},\mathbf{Z})& =E[\mathbf{Z}{\mathbf{Z}}^T]-E[\mathbf{Z}]E[\mathbf{Z}{]}^T\\ & ={\mathbf{I}}_n\\ \text{as}E[\mathbf{Z}{\mathbf{Z}}^T{]}_{ij}& =Cov(Z_i,Z_j)=0\text{independence}\\ E[\mathbf{Z}{\mathbf{Z}}^T{]}_{ii}& =Cov(Z_i,Z_i)=Var(Z_i)=1\text{independence}\end{array}$$

The mgf $$\begin{array}{rl}M(\mathbf{t})& =E\left[e^{{\mathbf{t}}^T\mathbf{Z}}\right]=\exp \left(\frac{1}{2}{\mathbf{t}}^T\mathbf{t}\right)\end{array}$$ by independence.

Now, assume a real symmetric semippositive definite matrix $\mathrm{\Sigma }$ that represents a variance-covariance matrix. From spectral decomposition, there exists ${\mathrm{\Sigma }}^{1/2}$ such that ${\mathrm{\Sigma }}^{1/2}{\mathrm{\Sigma }}^{1/2}=\mathrm{\Sigma }$ (from linear algebra). Also, this matrix is itself symmetric.

Assume a constant vector $\mu$. Then, $$\begin{array}{rl}\mathbf{X}& ={\mathrm{\Sigma }}^{1/2}\mathbf{Z}+\mu \\ ⟹E[\mathbf{X}]& ={\mathrm{\Sigma }}^{1/2}E[\mathbf{Z}]+\mu =\mu \\ Cov(\mathbf{X},\mathbf{X})& =E[\mathbf{X}{\mathbf{X}}^T]-E[\mathbf{X}]E[\mathbf{X}{]}^T\\ & ={\mathrm{\Sigma }}^{1/2}Cov(\mathbf{Z}){\left({\mathrm{\Sigma }}^{1/2}\right)}^T={\mathrm{\Sigma }}^{1/2}{\mathbf{I}}_n{\left({\mathrm{\Sigma }}^{1/2}\right)}^T=\mathrm{\Sigma }\\ M(\mathbf{t})& =E[\exp \left({\mathbf{t}}^T({\mathrm{\Sigma }}^T\mathbf{Z}+\mu )\right)]\\ & =E[\exp \left({\mathbf{t}}^T{\mathrm{\Sigma }}^{1/2}\mathbf{Z}\right)]E[\exp \left({\mathbf{t}}^T\mu \right)]=E[\exp \left(\left({\mathbf{t}}^T{\mathrm{\Sigma }}^{1/2}\right)\mathbf{Z}\right)]\exp \left({\mathbf{t}}^T\mu \right)\\ & =\exp \left(\frac{1}{2}{\mathbf{t}}^T\mathbf{\Sigma }\mathbf{t}\right)\exp \left({\mathbf{t}}^T\mu \right)\\ & =\exp ({\mathbf{t}}^T\mu +\frac{1}{2}{\mathbf{t}}^T\mathrm{\Sigma }\mathbf{t})\end{array}$$

which must be familiar from the univariate case where mgf is $\exp (\mu t+1/2{\sigma }^2{t}^2)$.

The distribution of $\mathbf{X}$ can be obtained using the Jacobian $$\begin{array}{rl}\mathbf{J}& =\frac{d\mathbf{Z}}{d\mathbf{X}}=\left|{\mathrm{\Sigma }}^{-1/2}\right|\\ \mathbf{Z}& ={\mathrm{\Sigma }}^{-1/2}(\mathbf{X}-\mu )\end{array}$$ where the negative sign denotes the inverse.

$$\begin{array}{rl}{f}_{\mathbf{X}}(\mathbf{x})& ={\left(\frac{1}{2\pi }\right)}^{\frac{n}{2}}\exp (-\frac{1}{2}(()\mathbf{X}-\mu {)}^T{\left({\mathrm{\Sigma }}^{-1/2}\right)}^T{\mathrm{\Sigma }}^{1/2}(\mathbf{X}-\mu ))\left|{\mathrm{\Sigma }}^{-1/2}\right|\\ & ={\left(\frac{1}{2\pi }\right)}^{\frac{n}{2}}\frac{1}{\left|{\mathrm{\Sigma }}^{1/2}\right|}\exp (-\frac{1}{2}{(\mathbf{X}-\mu )}^T\mathrm{\Sigma }(\mathbf{X}-\mu ))\end{array}$$