8.3 What it means and doesn't mean

For all of these tests, we use the vocabulary of “statistical significance”, which needs to be further clarified.

Significance

There is a term used in the literature called statistical significance. Roughly it means a value that is very unlikely to be zero (see Table 1.1 on page 51), or in other words, the value of zero is not within the 95% percentile. This is within 2 standard deviations of the value, so the following estimated values are not statistically significant:

$5 ± 3$ - the two-deviation range is [-1,11] contains the value 0
$7 ± 4$
$minus 3 ± 2$

but the following are statistically significant:

$5 ± 2$ - the two-deviation range is [1,9] does not contain the value 0
$7 ± 3$
$minus 3 ± 1$

Statistical significance, at the very unlikely level (i.e. 95% percentile) is often used as a rough guideline to publish a positive effect.

**Table 8.1.** Rough guide for the conversion of deviations away from zero and the qualitative labels for probability values for being a *significant* deviation.
number of deviations away from zero \	term	probability
$1 σ$	slightly likely/likely	0.7 (i.e. 7/10)
$2 σ$	very likely	0.95 (i.e. 19/20)
$3 σ$	extremely likely	0.01 (i.e. 1/100)
$greater than 4 σ$	virtually certain	$greater than 999, 999 / 1, 000, 000$

An unintuitive consequence

One consequence of this is that two studies that display different magnitudes for a quantity may not be statistically significant in their difference. The following example is based on an example from Gelman and Hill's book on Data Analysis.¹ Say we have two measurements with means and standard deviations:

25 $\pm$ 10
10 $\pm$ 3

Further, let us suppose that we are interested in whether the measurements are zero or not. The first measurement shows a significant effect (the two-deviation range is [5,45] does not contain zero), and the second one does as well (the two-deviation range is [4,16] does not contain zero). The difference between them is

$\begin{matrix} (25 - 10) \pm \sqrt{10^{2} + 3^{2}} = 15 \pm 10.4 \end{matrix}$

which is not significant. One should be careful comparing the magnitudes and uncertainties of measurements!

Adapted from Statistical Inference for Everyone, by Brian Blais (Bryant University), licensed under CC BY-SA 4.0 (dual-licensed under the GNU FDL 1.2 or later; this adaptation uses the CC BY-SA grant). Changes were made; this adaptation is distributed under the same license. License: CC-BY-SA-4.0.