Chapter 7: Priors, Likelihoods, and Posteriors

In Chapter 6 (Introduction to Parameter Estimation on page 121) we estimated the chance, $\theta$, that a bent coin would come up heads by combining a uniform prior for $\theta$ (i.e. all possible values are a-priori equally likely) and a binomial likelihood (i.e. given $\theta$, what is the probability of the data). This resulted in a Beta distribution for the posterior probability for $\theta$.

Notice what the procedure of Bayes' Recipe is and how the Bayesian inference works here.

1. Specify the prior probabilities for the models being considered

   We want to estimate a quantity (which we label as $\theta$), but begin with absolutely no knowledge of its value - we have a uniform prior probability.

2. Write the top of Bayes' Rule for all models being considered

   We construct a model for how different possible values of $\theta$ influence the outcome - a model we call the likelihood. In the case of the bent coin, the likelihood model is a binomial model, and describes the probability of flipping heads or tails given how bent the coin is (i.e. given $\theta$).

3. Put in the likelihood and prior values
4. Add these values for all models
5. Divide each of the values by this sum, $K$, to get the final probabilities

   Once we observe data, we can combine the prior and the model or likelihood using the Bayes' recipe, and obtain the posterior distribution for the unknown value, $\theta$, giving us the probability for each value, now updated with our new observations.