7.3 The Normal Distribution - Estimating From Data

Estimating the mean, $\mu$, knowing the deviation, $\sigma$

Typically one is provided with a series of measurements of a quantity, and we want to estimate the value of that quantity, and have a description of our uncertainty in the estimate. In Chapter 9 (Applications of Parameter Estimation and Inference on page 165) we go through a number of detailed examples of this process. Here, we simply summarize the result. We are given:

1. A series of $N$ measurements, data=$\{x_1,x_2,x_3,\dots ,x_N\}$
2. The real deviation, $\sigma$
3. We are modeling the data as a true value, $\mu$, with uncertainty with a likelihood from the Normal distribution with known deviation, $\sigma$, as in $\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}(0,\sigma )$. Further, we assume independence between the measurements.

Since in this case we are given $\sigma$, we wish then to estimate the parameter $\mu$. The result will be a probability distribution over $\mu$, with a best (i.e. most probable) value and an uncertainty in that value. The result is that the distribution of $\mu$ is also a Normal distribution,

$$\begin{array}{c}P(\mu |\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a},\sigma )=\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}(\overline{x},\sigma /\sqrt{N})\end{array}$$

where the center value (and thus the most probable value of $\mu$) is given by the sample mean of the data.

Sample Mean The sample mean of a set of $N$ samples, $x_1,x_2,\cdots ,x_N$ is given by

$$\begin{array}{c}\overline{x}\equiv \frac{x_1+x_2+x_3+\cdots +x_N}{N}\end{array}$$

The uncertainty in $\mu$ is given by $\sigma /\sqrt{N}$. As a consequence, larger $N$ (i.e. more data points), makes us more confident in the particular estimate for $\mu$.

Estimate of location parameter $\mu$ given $N$ samples and $\sigma$, the known deviation In summary, the best estimate for the location parameter $\mu$ in the Normal distribution given a set of $N$ samples, $x_1,x_2,\cdots ,x_N$ is given by

$$\begin{array}{c}\widehat{\mu }=\frac{x_1+x_2+x_3+\cdots +x_N}{N}\pm \sigma /\sqrt{N}\end{array}$$

Estimating the mean, $\mu$, not knowing the deviation, $\sigma$

If we are not so fortunate to be given the deviation, as in the previous case, then this parameter too must be estimated from the data. As a first step we can estimate the deviation with the sample deviation.

Sample Deviation The sample deviation of a set of $N$ samples, $x_1,x_2,\cdots ,x_N$ is given by

$$\begin{array}{c}S\equiv \sqrt{\frac{1}{N-1}((x_1-\overline{x}{)}^2+(x_2-\overline{x}{)}^2+\cdots +(x_N-\overline{x}{)}^2)}\end{array}$$

Approximate estimate of location parameter $\mu$ and deviation $\sigma$ given $N$ samples The posterior probability for $\mu$ and $\sigma$ given a set of $N$ samples, $x_1,x_2,\cdots ,x_N$ can be approximated with

$$\begin{array}{c}P(\mu |\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a})\sim \mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}(\overline{x},S/\sqrt{N})\\ P(\sigma |\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a})\sim \mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}(S,S^2/\sqrt{(N-1)/3})\end{array}$$

which works well if we have many ($N>30$) data points.

With a smaller data set, the value of $S$ as an estimate for the deviation becomes too small. When the estimate for $\sigma$ is too small, then the result is claiming more confidence in the estimate of the mean, $\mu$, than is warranted. This discrepancy depends on the number of data points, and thus it makes sense that the proper distribution should depend on the number of data points, in addition to the sample mean and deviation. The proper, although less convenient, result is that the posterior probability for $\mu$ takes the form of the Student's $t$ distribution,

Estimate of location parameter $\mu$ given $N$ samples and unknown $\sigma$ The posterior probability for $\mu$ takes the form of the Student's $t$ distribution,

$$\begin{array}{c}P(\mu |\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a})={\mathrm{S}\mathrm{t}\mathrm{u}\mathrm{d}\mathrm{e}\mathrm{n}\mathrm{t}}_{\mathrm{d}\mathrm{o}\mathrm{f}=N-1}(\overline{x},S/\sqrt{N})\end{array}$$

This distribution requires three numbers to specify, referred to as the mean ($\mu$), deviation ($\sigma$) and the degrees of freedom (dof). The degrees of freedom is defined in this case to be the number of data points less one, $N-1$.

Adapted from Statistical Inference for Everyone, by Brian Blais (Bryant University), licensed under CC BY-SA 4.0 (dual-licensed under the GNU FDL 1.2 or later; this adaptation uses the CC BY-SA grant). Changes were made; this adaptation is distributed under the same license. License: CC-BY-SA-4.0.