8.2 z-test

The $z$-test is the simplest test to use, and is perhaps the most common. It is used when we have the following assumptions:

1. We are modeling the data as a true value, $\mu$, with uncertainty
2. We are modeling the as a Normal distribution with known deviation, $\sigma$, as in ${\rm Normal}(0,\sigma)$.
3. We are assuming independence between the measurements.

The model of the data is

$$\begin{eqnarray*} {\rm data} = \mu + \text{uncertainty with probability Normal($\mu=0$,known $\sigma$)} \end{eqnarray*}$$

where $\mu$ represents the “true” value. The posterior distribution for $\mu$ also follows a Normal distribution, with a smaller uncertainty, $\sigma/\sqrt{N}$ where $N$ is the number of data points.

To use the $z$-test, we perform the following steps:

1. Calculate our best estimate for $\mu$, denoted as $\hat{\mu}$.
2. Given the known uncertainty, $\sigma$ of a single measurement, determine the range of credible values for $\mu$ within the uncertainty of the estimate for the $N$ observations, $\sigma/\sqrt{N}$.
3. Test to see if the credible range includes zero.
4. If so, then the test passes, and we can be reasonably confident that the parameter is non-zero - that the effect is real.
5. If the test fails, i.e. the credible range does not include zero, then under the model the possibility of a zero-effect cannot be reasonably excluded.

There are several scenarios where we use the $z$-test, each with the same procedure, differing only in the method of estimating the “true” value $\mu$.

1. For $N$ independent observations, $x_{1}, x_{2}, …, x_{N}$ we have the best estimate given by the sample mean, and uncertainty related to the single-measurement deviation, $\sigma$, as

   $$\begin{eqnarray*} \hat{\mu} = \frac{x_{1} + x_{2} + \cdots + x_{N}}{N} \pm \sigma/\sqrt{N} \end{eqnarray*}$$

2. When estimating a proportion, for a large number of events $N$ of which a fraction $f\equiv h/N$ are successful, we have

   $$\begin{eqnarray*} \hat{\mu} &\approx& f \\ \sigma/\sqrt{N} &\approx& \sqrt{f(1-f)/N} \end{eqnarray*}$$

3. For smallish data sets, $5<N<30$, where the uncertainty is not known,

   $$\begin{eqnarray*} \hat{\mu} &\approx& \frac{x_{1} + x_{2} + \cdots + x_{N}}{N}\\ \sigma/\sqrt{N} &\approx& k S/\sqrt{N} \end{eqnarray*}$$

   where we replace the known $\sigma/\sqrt{N}$ from the previous case with an estimate using the sample standard deviation and an adjustment for small data set parameter $k$,

   $$\begin{eqnarray*} S^{2}&=&\frac{1}{N-1}\left( (x_{1}-\bar{x})^{2}+\cdots+(x_{N}-\bar{x})^{2}\right) \\ k&\equiv& 1+\frac{20}{N^{2}} \end{eqnarray*}$$

Adapted from Statistical Inference for Everyone, by Brian Blais (Bryant University), licensed under CC BY-SA 4.0 (dual-licensed under the GNU FDL 1.2 or later; this adaptation uses the CC BY-SA grant). Changes were made; this adaptation is distributed under the same license. License: CC-BY-SA-4.0.