6.7 Best Estimates

Perhaps surprisingly, there is not a single answer to the best estimate for $θ$ given the posterier distribution, like the one shown in Figure 6.9. There are several plausible measures, each with their own advantages. Any specific estimate of a parameter (e.g. $θ$ ) is denoted with a hat (e.g. $\hat{θ}$ ) in the descriptions that follow.

The Mode Also known as the maximum a-posteriori probability (MAP) estimate, the mode is the maximum of the posterior probability. In the case of a Beta distribution with $h$ successes in $N$ trials, we have

$\begin{matrix} {\hat{θ}}_{m o d e} = \frac{h}{N} \end{matrix}$

The Mean Also known as the expected value or average value, the mean of a distribution of a parameter $θ$ is defined to be the sum of all of the possible values of $θ$ times the posterior probability of $θ$ ,

$\begin{matrix} {\hat{θ}}_{m e a n} = \sum_{θ} θ \times P (θ | d a t a) \end{matrix}$

It is one measure of the middle of the distribution. In the special case of a Beta distribution with $h$ successes in $N$ trials, we have

$\begin{matrix} {\hat{θ}}_{m e a n} = \frac{h + 1}{N + 2} \end{matrix}$

Intuitively this is the same as the MAP of the Beta distribution, with one more success and one more failure than actually observed. Further, for the Beta distribution, the mean value ${\hat{θ}}_{m e a n}$ represents the predictive probability of a successful event on the next observation.

Note

The Mean

Also known as the expected value, the mean of a distribution of a parameter $θ$ is defined to be the sum of all of the possible values of $θ$ times the posterior probability of $θ$ , as in

$\begin{matrix} {\hat{θ}}_{m e a n} = \sum_{θ} θ \times P (θ | d a t a) \end{matrix}$

It is one measure of the middle of the distribution.

The Median Also known as the 50%-percentile, the median represents the middle of the distribution such that the probability of the parameter below the median equal to the probability of the parameter above the median.

$\begin{matrix} P (θ \leq {\hat{θ}}_{m e d i a n} | d a t a) = P (θ \geq {\hat{θ}}_{m e d i a n} | d a t a) = 0.5 \end{matrix}$

Note

The Median

Also known as the 50%-percentile, the median represents the middle of the distribution such that the probability of the parameter below the median equal to the probability of the parameter above the median.

$\begin{matrix} P (θ \leq {\hat{θ}}_{m e d i a n} | d a t a) = 0.5 \\ P (θ \geq {\hat{θ}}_{m e d i a n} | d a t a) = 0.5 \end{matrix}$

“Assume 2 successes and 2 failures” median approximation For the Beta distribution there is no simple form for the median, but a decent approximation which we will use is given by¹

$\begin{matrix} {\hat{θ}}_{m e d i a n} \approx \frac{h + 2}{N + 4} \end{matrix}$

Intuitively this is the same as the MAP of the Beta distribution, with two more successes and two more failures than actually observed, and is thus referred to as the “Assume 2 successes and 2 failures” median approximation.

Note

“Assume 2 successes and 2 failures” median approximation

For the Beta distribution there is no simple form for the median, but a decent approximation which we will use is given by

$\begin{matrix} {\hat{θ}}_{m e d i a n} \approx \frac{h + 2}{N + 4} \end{matrix}$

Although each of these has their advantages, most notably ease of computation (especially for the mode and the mean), we will typically use the median of the distribution as the best estimate for the following reasons:

the median is intuitive as literally the middle of the distribution
the median is not as sensitive to distributions that are highly asymmetric

In most practical examples it may not make much difference, and for some distributions (such as the Normal distribution described in Chapter 7 (Priors, Likelihoods, and Posteriors)) there is not difference - the mean is the median which is also the mode.

Example 1 What is the best estimate of the probability of a bent coin flipping heads, given the observation of 9 tails and 3 heads?

Show solution

If we take the best estimate to be the median, then we have from the “assuming 2 successes and 2 failures” method,

$\begin{matrix} {\hat{θ}}_{m e d i a n} \approx \frac{h + 2}{N + 4} \\ = \frac{5}{16} = 0.313 \end{matrix}$

Notice that the maximum probability was at the somewhat lower value

$\begin{matrix} {\hat{θ}}_{m o d e} = \frac{h}{N} = \frac{3}{12} = 0.25 \end{matrix}$

One reason why the median is a better estimate in this case is because, as shown in Figure 6.9, there is more probability (i.e. area under the curve) to the right of the maximum than to the left, so the best estimate should be greater than the one given by the mode.

The same posterior with all three estimates marked: the mode h/(h+t) in red, the mean (h+1)/(h+t+2) in green, and the 'assume 2 successes and 2 failures' median (h+2)/(h+t+4) in purple. At the section's own example, 3 heads and 9 tails, they sit at 0.25, 0.286 and 0.313 - visibly apart, and in that order, because the curve is skewed to the right exactly as the text says. Drag h up toward t and watch the three markers converge; at h = t they are the same line, which is the section's closing remark that for a symmetric distribution the mean IS the median IS the mode and the choice stops mattering. Then push the data the other way - h = 1, t = 20 - and the spread between them opens to more than a tenth of the whole range, which is when the choice matters most and when the median's insensitivity to the long tail earns its keep.

Adapted from Statistical Inference for Everyone, by Brian Blais (Bryant University), licensed under CC BY-SA 4.0 (dual-licensed under the GNU FDL 1.2 or later; this adaptation uses the CC BY-SA grant). Changes were made; this adaptation is distributed under the same license. License: CC-BY-SA-4.0.