Login
📚 Statistical Inference for Everyone
Chapters ▾
1 Introduction to Probability2 Applications of Probability3 Random Sequences and Visualization4 Introduction to Model Comparison5 Applications of Model Comparison6 Introduction to Parameter Estimation7 Priors, Likelihoods, and Posteriors8 Common Statistical Significance Tests9 Applications of Parameter Estimation and Inference10 Multi-parameter Models11 Introduction to MCMC12 Concluding Thoughts13 Appendix A: Computational Analysis14 Appendix B: Notation and Standards15 Appendix C: Common Distributions and Their Properties16 Appendix D: Tables

6.4 Moving Toward the Continuous

There is a practical problem that we face at this point, when we consider a generic bent coin. Perhaps it doesn't fit in one of the 11 models considered, falling somewhere in between, for example with P(heads)=0.132464. Ones' first thought might be to include one thousand coins or one million coins instead of the 11 we've considered so far, so we could have coin 132464, coin 132465, coin 132466, etc... Although this can be done, we run into two problems

  1. Because we are dealing with so many models, the probability associated with any single model gets very small - and gets smaller with the more models you consider
  2. We can't practically distinguish between models such as P(heads)=0.132464 and P(heads)=0.132465 (the last digit is different here)

In order to solve both of these problems mathematically, we introduce the concept of a continuous distribution. We start by labeling the model with a continuous number rather than an integer. In our present case it makes sense to label the model with the probability that the coin flips heads. We'll call this label θ, and it will have a value between 0 (heads are impossible) and 1 (heads are certain) and can take on any value in between. Because we now have an infinite number of labels, we have two consequences:

  1. We can't simply add up all the probabilities to get our value of K to make everything add up to 1. Instead, we look at areas under the curve and make sure the entire area equals 1.
  2. Because, with distributions, areas under the curve (and not the values of the distribution itself) are the probabilities, we can only speak about ranges of values. For example, we can speak meaningfully about the probability of θ between 0.3 and 0.4 (i.e. P(0.3<θ<0.4)). When we write down something like P(θ)=1 we're not talking about a probability of a single label but rather the magnitude of the distribution at that label, θ.

We revisit Bayes' Recipe again, using the distributions. This time we also will look at pictures of the distributions as we progress.

Function graph showing y = (x^h*(1-x)^t)/((h/(h+t))^h*(1-h/(h+t))^t) on x in [0, 1] with the region between the curve and the x-axis shaded and y = exp(-((x-h/(h+t))/0.005)^2) on x in [0, 1]. Adjustable parameters: Heads observed (h) = 3, Tails observed (t) = 9. Viewing window: x from -0.07 to 1.13, y from -0.13 to 1.07.
The posterior for theta after h heads and t tails, scaled so its peak is 1 - heights on this curve are not probabilities, only areas under it are, which is the point the section is making. The dashed vertical line marks the maximum, which always lands exactly at h/(h+t). Set h = 0 and t = 0 and the curve is flat: that is the uniform prior, the first of the three pictures beside this text. Set h = 3, t = 9 and it is the third one, peaking at 0.25 with a long tail to the right. Everything the chapter says next can be read off this one figure by dragging: adding a tail slides the peak left, adding a head slides it right, and holding the RATIO fixed while raising both - 3 and 9, then 6 and 18, then 9 and 27 - leaves the peak at 0.25 but pulls the curve in around it, which is what 'more data sharpens the estimate' actually looks like. Push t to 30 with h at 0 and the curve piles against theta = 0 without ever reaching it: no amount of tails makes heads logically impossible.
  1. Specify the prior probabilities for the models being considered:

    P(θ)=1.

    Plot of a flat prior P(θ)=1 across θ from 0 to 1, drawn as a horizontal line at height 1 with the region underneath shaded and annotated 'Area Under Curve = 1'.
  2. Write the top of Bayes' Rule for all models being considered:

    We can write one equation for all of the models labeled by θ at once as

    P(θ|data=9T,3H)∼P(data=9T,3H|θ)P(θ).

  3. Put in the likelihood and prior values.

    We use the binomial model, one equation for all models, remembering that for a model labeled by θ the probability for that coin flipping heads is P(heads)=θ. Thus we get the likelihood and prior values as

    P(θ|data)∼P(data|θ)·P(θ)P(θ|data9T,3H)∼(123)θ3×(1−θ)9·1

    Skewed curve proportional to the posterior for θ given 9 tails and 3 heads, peaking near θ=0.25 at a height of only about 0.26; the shaded region is annotated 'Area Under Curve NOT 1' — the curve is not yet normalized.
  4. Find the area under this curve, and call it K.
  5. Divide each of the values of the curve by this are, K, to get the final probabilities where the area under the curve is 1.
    The same posterior curve for θ given 9 tails and 3 heads after normalization: it peaks near θ=0.25 at a height of about 3.3 and its shaded area is annotated 'Area Under Curve = 1'.

Usually these steps are done for you, for a specific data set, and you are given the final posterior distribution to use in answering any questions. However, for any particular case it is important to know what assumptions have been made in the choice of models and model parameters.

Adapted from Statistical Inference for Everyone, by Brian Blais (Bryant University), licensed under CC BY-SA 4.0 (dual-licensed under the GNU FDL 1.2 or later; this adaptation uses the CC BY-SA grant). Changes were made; this adaptation is distributed under the same license. License: CC-BY-SA-4.0.