1.5 Rules of Probability

From the rule for mutually exclusive events (Equation 1.2), we assign the following probabilities for the first draw from this deck:

$P (10) = \frac{4}{52}$
$P (♥) = \frac{13}{52} = \frac{1}{4}$
$P (10 ♥) = \frac{1}{52}$
$P (face card) = \frac{12}{52}$
$P (number card) = \frac{40}{52}$

It turns out that mathematically, the rules for fractions of things and of probabilities are the same. Thus, to gain an understanding for the rules of probability, we will calculate fractions (which are more immediately intuitive), and then summarize the same rule for probabilities.

Negation Rule

In this section I'll use the letter $F$ for fraction, and we can determine the values simply by counting. The fraction of cards which are hearts ( $♥$ ) is

Note

Either-or fallacy. The negation rule, should not be taken to imply that everything is “black and white,” or “there are only two sides to every story.” It really is just a statement of logic, should be carefully considered and has some limitations. For example, the following is true,

$\begin{matrix} P (object is black) + P (object is not black) = 1 \end{matrix}$

However, this does not mean the same thing as

$P open parenthesis object is black close parenthesis plus P open parenthesis object is white close parenthesis not equal to 1$

“Not black” is not the equivalent of “white.” It could be red, or gray, or some other color. A common logical fallacy sometimes referred to as the “either-or fallacy” or the “fallacy of the excluded middle,” turns on this point. Some examples of these fallacies are:

If we don't reduce public spending, our economy will collapse.
You're either with us or you're a terrorist.
Either modern medicine can explain how Ms. X was cured, or it is a miracle.

$\begin{matrix} F (♥) = \frac{13}{52} = \frac{1}{4} \end{matrix}$

The fraction of cards which are not hearts (i.e. the 3 other suits) is:

$\begin{matrix} F (not ♥) = \frac{13 \times 3}{52} = \frac{3}{4} \end{matrix}$

These numbers add up to one: $F (♥) + F (not ♥) = 1$ . We can do this with more complex statements.

$\begin{matrix} F (first card is a face card) = \frac{12}{52} \\ F (first card is not a face card) = \frac{40}{52} \\ F (first card is a face card) + F (first card is not a face card) = 1 \end{matrix}$

Example 1 What is the fraction of the first card as a jack given that we know that the first card is a face card?

Show solution

We can also apply the negation rule to conditional statements, like “the first card is a jack given that we know that the first card is a face card.” Notice that there are 12 cards that are face cards, so we restrict our counts to those.

$\begin{matrix} F (jack | face card) = \frac{4}{12} = 1 / 3 \\ F (not a jack | face card) = \frac{8}{12} = 2 / 3 \\ F (jack | face card) + F (not a jack | face card) = 1 \end{matrix}$

and they add up to one.

Negation Rule Given any information, we have

$\begin{matrix} P (statement | information) + P (not statement | information) = 1 \end{matrix}$

$\begin{matrix} P (A | B) + P (not A | B) = 1 & (1.7) \end{matrix}$

Product Rule

The product rule comes from looking at the combination of events: event A and event B. As before, we'll work on the numbers from the fractions of the card game.

Example 2 What is the fraction of cards that are Jacks and a heart?

Show solution

This is clearly $F open parenthesis J ♥ close parenthesis equals 1 / 52$ , but we can look at it a different way that is equivalent. We note that the Jacks constitute 4/52 of the cards, and that of those 4, only one quarter of them are hearts (one card out of the four cards). So, we can arrive at the fraction of J $♥$ by taking one quarter of the fraction of jacks. So what we have is

$\begin{matrix} F (jack and ♥) = F (♥ | jack) \times F (jack) = \frac{1}{4} \times \frac{4}{52} = \frac{1}{52} \end{matrix}$

One can equivalently reason from the suit first: the hearts constitute 13/52 of the cards, and that of those 13, the Jacks constitute 1/13 of the cards. So, we can arrive at the fraction of J $♥$ by taking one thirteenth of the fraction of $♥$ . Again, we have

$\begin{matrix} F (jack and ♥) = F (jack | ♥) \times F (♥) = \frac{1}{13} \times \frac{13}{52} = \frac{1}{52} \end{matrix}$

In general we have

$\begin{matrix} P (A and B) = P (A | B) P (B) = P (B | A) P (A) & (1.8) \end{matrix}$

Product Rule

Note

Product Rule

$\begin{matrix} P (A and B) = P (A | B) P (B) \\ = P (B | A) P (A) \end{matrix}$

Example 3 What is the probability of drawing two Kings in a row?

Show solution

This is the same as

$P open parenthesis K sub 2 and K sub 1 close parenthesis$

From the product rule (Equation 1.8) we have

$P open parenthesis K sub 2 and K sub 1 close parenthesis equals P open parenthesis K sub 2 vertical bar K sub 1 close parenthesis P open parenthesis K sub 1 close parenthesis$

The second part is straight forward: $P open parenthesis K sub 1 close parenthesis equals 4 / 52$ . The first part is asking the probability of drawing a second king, knowing that we have drawn a king on the first draw. Now, there are only 51 cards remaining when we do the second draw, and only 3 kings. Thus, we have $P open parenthesis K sub 2 vertical bar K sub 1 close parenthesis equals 3 / 51$ and finally

$\begin{matrix} P (K_{2} and K_{1}) = P (K_{2} | K_{1}) P (K_{1}) \\ = \frac{3}{51} \times \frac{4}{52} = \frac{1}{221} \end{matrix}$

Independence

As a specific case of the product rule, we can change the rule of the card games such that we reshuffle the deck after each draw. In this way, the result of one draw gives you no information about other draws. In this case, the events are considered independent.

Independent Events Two events, A and B, are said to be independent if knowledge of one gives you no information on the other. Mathematically, this means

$\begin{matrix} P (A | B) = P (A) \end{matrix}$

and

$\begin{matrix} P (B | A) = P (B) \end{matrix}$

In this case, the product rule reduces to the simplified rule for independent events: the product of the individual event probabilities.

$\begin{matrix} P (A and B) = P (A) \times P (B) & (1.9) \end{matrix}$

Joint Probabilities for Independent Events

We have already seen an example of this, when we looked at drawing the Jack of Hearts: drawing a heart gives you no information about whether it is a jack, and vice versa. Thus,

$\begin{matrix} P (♥ | jack) = P (♥) \end{matrix}$

Conjunction

One of the consequences of combinations of events is that the probability of two events happening, A and B, has to be less than (or possibly equal to) the probability of just one of them, say A, happening. The mathematical fact is seen by looking at the magnitude of the terms in the product rule

$\begin{matrix} P (A and B) = \underset{less than or equal to 1}{\underset{⏟}{P (B | A)}} \times P (A) \leq P (A) \end{matrix}$

In other words, coincidences are less likely than either event happening individually. We intuitively know this, when we make comments like “Wow! What are the chances of that?” referring to, say, someone winning the lottery and then getting struck by a car the next day. Sometimes, however, it seems as if one's intuition does not match the conclusions of the rules of probability. One such case is called the conjunction fallacy.

In an interesting experiment, Tversky and Kahneman¹ gave the following survey:

85% chose option 2.² This, they attributed, to the conjunction fallacy - mistaking the conjunction of two events as more probable than a single event. They went further and did a survey of medical internists with the following

Which is more likely: the victim of an embolism (clot in the lung) will experience partial paralysis or that the victim will experience both partial paralysis and shortness of breath?

and again, 91 percent of the doctors chose that the clot was less likely to cause the rare paralysis rather than to cause the combination of the rare paralysis and the common shortness of breath.

Even when correct, the consequence for conjunctions can be misused, or at least misidentified. Returning to our example of someone winning the lottery and then getting struck by a car the next day, rare events occur frequently, as long as you have enough events. There are millions of people each day playing the lottery, and millions getting struck by cars each day. We will explore this problem later in Section 2.6, but one immediate consequence is that winning the lottery and getting struck by a car the next day probably happens somewhere fairly regularly.

Sum Rule

Now we consider the statements of the form A or B. For example, in the card game, what is the fraction of cards that are jacks or are hearts. By counting we get the 13 hearts and 3 more jacks that are not contained in the 13 hearts, or $F (jack or ♥) = \frac{13 + 3}{52} = 16 / 52$ . Now, if we tried to separate the terms, and do:

$\begin{matrix} F (jack) + F (♥) = \frac{4}{52} + \frac{13}{52} = \frac{17}{52} \end{matrix}$

then we get a number that is too big! It is too big because we've double-counted the jack of hearts. Adjusting for this, by subtracting one copy of this fraction, we get

$\begin{matrix} F (jack) + F (♥) - F (jack and ♥) = \frac{4}{52} + \frac{13}{52} - \frac{1}{52} = \frac{16}{52} = F (jack or ♥) \end{matrix}$

In general

$\begin{matrix} P (A or B) = P (A) + P (B) - P (A and B) & (1.10) \end{matrix}$

Sum Rule

Sum Rule for Exclusive Events If two events are mutually exclusive the sum rule reduces to

$\begin{matrix} P (A or B) = P (A) + P (B) & (1.11) \end{matrix}$

because $P open parenthesis A and B close parenthesis equals 0$ for such events.

So the probability of rolling a 1 or a 2 on one die is 2/6.

One more variant on the Sum Rule is where we have 3 propositions. It can be a bit tedious to write it all out, but the end result looks a lot like the original Sum Rule. All we do is break up the terms in pieces, and then apply the Sum Rule to each piece.

$\begin{matrix} P (A or B or C) = P (A or [B or C]) \\ = P (A) + P (B or C) - P (A and [B or C]) \\ = P (A) + P (B) + P (C) - P (B and C) - \\ P (A and B or A and C) \\ = P (A) + P (B) + P (C) - P (B and C) - \\ [P (A and B) + P (A and C) - \\ P (A and B and A and C)] \end{matrix}$

which leads finally to

$\begin{matrix} P (A or B or C) = P (A) + P (B) + P (C) - \\ P (A and B) - P (B and C) - P (A and C) + \\ P (A and B and C) & (1.12) \end{matrix}$

Sum Rule for Three Events

Note

Sum Rule for Three Events

$\begin{matrix} P (A or B or C) = P (A) + P (B) + P (C) - \\ P (A and B) - \\ P (B and C) - \\ P (A and C) + \\ P (A and B and C) \end{matrix}$

In words, when you're looking for the sum of several events, we add the probabilities (i.e. $P (A) + P (B) + P (C)$ ), then subtract the double counting (i.e. $P (A and B)$ ) as before. Finally, we need to add back in the triple count (i.e. $P (A and B and C)$ ) because it was taken out too many times with the double count. The accounting here can be somewhat prone to error, but the concepts are always the same: when you add probabilities of events, say $A$ and $B$ , together the term $P (A)$ includes the probability of both $P (A and B)$ and the term $P (B)$ includes the probability of both $P (A and B)$ , so you've included that probability twice and need to subtract one of them to balance the books. Likewise (although it is harder to show), the first six terms in Equation 1.12 end up subtracting one too many copies of $P (A and B and C)$ , and we need to add one in at the end.

Marginalization

Another consequence of the sum rule and the product rule is a process called marginalization.

Example 5 Marginalization and Card Suit

Imagine we have a number of conditional statements, like:

$\begin{matrix} P (jack | ♥) = \frac{1}{13} \\ P (jack | ♦) = \frac{1}{13} \\ P (jack | ♠) = \frac{1}{13} \\ P (jack | ♣) = \frac{1}{13} \end{matrix}$

but we are interested in just the probability of drawing a jack, regardless of the suit, or in our notation

$\begin{matrix} P (jack) \end{matrix}$

The marginalization procedure for this problem looks like:

$\begin{matrix} P (jack) = \overset{all possibilities}{\overset{⏞}{P (jack | ♥) \times P (♥) +}} \\ P (jack | ♦) \times P (♦) + \\ P (jack | ♠) \times P (♠) + \\ P (jack | ♣) \times P (♣) \\ = \frac{1}{13} \times \frac{1}{4} + \frac{1}{13} \times \frac{1}{4} + \frac{1}{13} \times \frac{1}{4} + \frac{1}{13} \times \frac{1}{4} \\ = \frac{4}{52} \end{matrix}$

Marginalization If we have a complete set of conditional statements, like

$P open parenthesis A vertical bar B sub 1 close parenthesis; P open parenthesis A vertical bar B sub 2 close parenthesis; P open parenthesis A vertical bar B sub 3 close parenthesis; P open parenthesis A vertical bar B sub 4 close parenthesis; ⋮$

then the unconditional probability is found by marginalizing over all possible values of the conditional events, like

$\begin{matrix} P (A) = \overset{all possible B s}{\overset{⏞}{P (A | B_{1}) P (B_{1}) + P (A | B_{2}) P (B_{2}) + P (A | B_{3}) P (B_{3}) + \dots}} & (1.13) \end{matrix}$

Note

Marginalization

If we have a complete set of conditional statements, like

$P open parenthesis A vertical bar B sub 1 close parenthesis, P open parenthesis A vertical bar B sub 2 close parenthesis, P open parenthesis A vertical bar B sub 3 close parenthesis, P open parenthesis A vertical bar B sub 4 close parenthesis, ⋯$

then the unconditional probability is found by marginalizing over all possible values of the conditional events, like

$\begin{matrix} P (A) = \overset{all possible B s}{\overset{⏞}{P (A | B_{1}) P (B_{1}) + P (A | B_{2}) P (B_{2}) + \dots}} \end{matrix}$

Bayes' Rule

One of the most consequential rules of probability is what is known as Bayes' Rule, sometimes called Bayes' Theorem. We will use this rule throughout this book, and see its many applications. It comes as a direct result of the product rule (Equation 1.8)

$\begin{matrix} P (A and B) = P (A | B) P (B) = P (B | A) P (A) \end{matrix}$

Rearranging, we get

$\begin{matrix} P (A | B) = \frac{P (B | A) P (A)}{P (B)} & (1.14) \end{matrix}$

Bayes' Rule

We can verify this again with the intuitions we have in the simple card game.

Example 6 What is the probability of drawing a jack, knowing that you've drawn a face card?

Show solution

In terms of fractions, this should be $F (jack | face card) = 4 / 12 = 1 / 3$ . Applying Bayes' Rule to the fractions we get:

$\begin{matrix} F (jack | face) = \frac{F (face | jack) \times F (jack)}{F (face)} \\ = \frac{\frac{4}{4} \times \frac{4}{52}}{\frac{12}{52}} = \frac{4}{12} = \frac{1}{3} \end{matrix}$

Although this calculation is true, it isn't particularly enlightening. It is nicer to cast the problem back into probability terms, rather than fractions, and compare the probability of drawing a jack to the probability of the same thing (i.e. drawing a jack) given that we know that we've drawn a face card. This is

$\begin{matrix} P (jack) = \frac{1}{13} \\ P (jack | face card) = \frac{1}{3} \end{matrix}$

This comparison highlights what Bayes' Rule represents: learning. When you are asked what the probability of drawing a jack, from the knowledge of the simple card game, you calculate the value of 1/13. Once you learn that you drew a face card, you update your knowledge to include that information, and modify your probability assignments to reflect this. This leads to an increased chance of the card being a jack.

In a nutshell, Bayes' Rule represents learning:

$\begin{matrix} Initial Belief + New Data \to Improved Belief \end{matrix}$

It is used in science to infer causes from effects, and can thus be written

$\begin{matrix} P (cause | effect) = \frac{P (effect | cause) \times P (cause)}{P (effect)} \end{matrix}$

To infer the probability of a particular cause, given the events you observe in the world, you first have to know the probability of the cause itself (i.e. rarer causes will reduce the prior probability), and how likely that the cause you're looking at could have produced the effects you've observed. These two items are the $P (cause)$ and $P (effect | cause)$ terms, respectively. The entire calculation is scaled by $P (effect)$ which is all of the other ways that the effects could have been produced by other causes.. Thus, it is not enough to show that giving a particular medicine is followed by the symptoms disappearing to establish that the medicine was the likely cause of the symptoms disappearing. You have to calculate what other possible causes could have had those effects, such as the normal functioning of the immune system or the placebo effect. This is why carefully controlled studies are necessary, to eliminate all of the other possible causes and to determine the true cause of the effects observed.

We will spend large portions of several chapters on Bayes' Rule, to explore its long-ranging consequences.

Adapted from Statistical Inference for Everyone, by Brian Blais (Bryant University), licensed under CC BY-SA 4.0 (dual-licensed under the GNU FDL 1.2 or later; this adaptation uses the CC BY-SA grant). Changes were made; this adaptation is distributed under the same license. License: CC-BY-SA-4.0.