1.3 What is Probability?

Probability theory is nothing but common sense reduced to calculation. - Laplace

When you think about probability, the first things that might come to mind are coin flips (“there's a 50-50 chance of landing heads”), weather reports (“there's a 20% chance of rain today”), and political polls (“the incumbent candidate is leading the challenger 53% to 47%”). When we speak about probability, we speak about a percentage chance (0%-100%) for something to happen, although we often write the percentage as a decimal number, between 0 and 1. If the probability of an event is 0 then it is the same as saying that you are certain that the event will never happen. If the probability is 1 then you are certain that it will happen. Life is full of uncertainty, so we assign a number somewhere between 0 and 1 to describe our state of knowledge of the certainty of an event. The probability that you will get struck by lightning sometime in your life is $p=0.0002$, or 1 out of 5000. Statistical inference is simply the inference in the presence of uncertainty. We try to make the best decisions we can, given incomplete information.

One can think of probability as a mathematical short-hand for the common sense statements we make in the presence of uncertainty. This short-hand, however, becomes a very powerful tool when our common sense is not up to the task of handling the complexity of a problem. Thus, we will start with examples that will perhaps seem simple and obvious, and move to examples where it would be a challenge for you to determine the answer without the power of statistical inference.

Let's walk through a simple set of examples to establish the notation, and some of the basic mathematical properties of probabilities.

Card Game

A simple game can be used to explore all of the facets of probability. We use a standard set of cards (Figure 1.1) as the starting point, and use this system to set up the intuition, as well as the mathematical notation and structure for approaching probability problems.

We start with what I simply call the simple card game, which goes like:

simple card game ≡ From a standard initially shuffled deck, we draw one card, note what card it is and set it aside. We then draw another card, note what card it is and set it aside. Continue until there are no more cards, noting each one along the way.

There are certain principles that guide us in developing the mathematical structure of probability. We start with some common sense notions, written in English, and then write them as general principles. These principles, then, constrain our mathematics so that we can apply the ideas quantitatively.

When asked “what is the probability of drawing a red on the first draw?” you would generally say 50-50, or 50%, or equivalently written as a probability, $P(R_1)=0.5$. The reason for this is that we are completely ignorant of the initial conditions of the deck (i.e. where each card is located in the deck after the initial shuffling). Given this level of (or lack of) knowledge, we could swap the colors of the two suits and we would have an equivalent state of knowledge - the problem would be identical. We will keep coming back to this concept, but in general:

Principle of Knowledge and Probability Equivalent states of knowledge must yield equivalent probability assignments.

Because of this principle, we are led to the conclusion that

$$\begin{array}{c}P(R_1)=P(B_1)\end{array}$$

where $R_1$ represents the statement “a red on the first draw” and $B_1$ represents “a black on the first draw.” Because these are the only two options, and they are mutually exclusive, then they must add up to 1. Thus we have

$$\begin{array}{c}P(R_1)=1-P(B_1)\end{array}$$

which leads directly to our original assignment

$$\begin{array}{c}P(R_1)=P(B_1)=0.5\end{array}$$

Mutually Exclusive If I have a list of mutually exclusive events, then that means that only one of them could possibly be true. Example events include flipping heads or tails with a coins, rolling a 1, 2, 3, 4, 5 or 6 on dice, or drawing a red or black card from a deck of cards. In terms of probability, this means that, for events A and B, $P(A\mathbf{\text{and }}B)=0$.

Non Mutually Exclusive If I have a list of events that are not mutually exclusive, then it is possible for two or more to be true. Examples include weather with rain and clouds or holding the high and the low card in a poker game.

Now, this was a long-winded way to get to the answer we knew from the start, but that is how it must begin. We start working things out where our common sense is strong, so that we know we are proceeding correctly. We can then, confidently, apply the tools in places where our common sense is not strong.

In summary, with no more information than that there are two mutually exclusive possibilities, we assign equal probability to both. If there are only two colors of cards in equal amounts, red and black, then the probability of drawing a red is $P(R_1)=0.5$ and the probability for a black is the same, $P(B_1)=0.5$.

Other Observations

If instead of just the color, we were interested in the suit (hearts, diamonds, spades, and clubs), then there would be four equal and mutually exclusive possibilities. We have a certain number of possibilities, and our state of knowledge is exactly the same if we simply swap around the labels on the cards. If we're interested in the specific card, not just the suit, the logic is the same. Thus, we have

$$\begin{array}{c}P(♠)=P(♣)=P(♦)=P(♥)\end{array}$$

and for drawing one specific card from the deck,

$$\begin{array}{c}P(A♠)=P(2♠)=P(3♠)=\cdots =P(K♥)\end{array}$$

Further, they all must add up to 1, so we get for suits

$$\begin{array}{c}P(♠)+P(♣)+P(♦)+P(♥)=1\end{array}$$

and for the specific card from the deck,

$$\begin{array}{c}\underset{\text{52 cards}}{\underbrace{P(A♠)+P(2♠)+P(3♠)+\cdots +P(K♥)}}=1\end{array}$$

Putting it together, we get for the suits

$$\begin{array}{c}P(♠)=P(♣)=P(♦)=P(♥)=\frac{1}{4}\end{array}$$

and for the specific card

$$\begin{array}{c}P(A♠)=P(2♠)=P(3♠)=\cdots =P(K♥)=\frac{1}{52}\end{array}$$

Probabilities for Mutually Exclusive Events In general, for mutually exclusive events, we have

$$\begin{array}{cc}P(A)=\frac{\text{(number of cases favorable to A)}}{\text{(total number of equally possible cases)}}& \text{(1.2)}\end{array}$$

Adapted from Statistical Inference for Everyone, by Brian Blais (Bryant University), licensed under CC BY-SA 4.0 (dual-licensed under the GNU FDL 1.2 or later; this adaptation uses the CC BY-SA grant). Changes were made; this adaptation is distributed under the same license. License: CC-BY-SA-4.0.