3.4 Probability Theory

Learning Outcomes

By the end of this section, you should be able to:

Describe the basic concepts of probability and apply these concepts to real-world applications in data science.
Apply conditional probability and Bayes’ Theorem.

Probability is a numerical measure that assesses the likelihood of occurrence of an event. Probability applications are ubiquitous in data science since many decisions in business, science, and engineering are based on probability considerations. We all use probability calculations every day as we decide, for instance, whether to take an umbrella to work, the optimal route for a morning commute, or the choice of a college major.

Basic Concepts of Probability

We have all used probability in one way or another on a day-to-day basis. Before leaving the house, you might want to know the probability of rain. The probability of obtaining heads on one flip of a coin is one-half, or 0.5.

A data scientist is in interested in expressing probability as a number between 0 and 1 (inclusive), where 0 indicates impossibility (the event will not occur) and 1 indicates certainty (the event will occur). The probability of an event falling between 0 and 1 reflects the degree of uncertainty associated with the event.

Here is some terminology we will be using in probability-related analysis:

An outcome is the result of a single trial in a probability experiment.
The sample space is the set of all possible outcomes in a probability experiment.
An event is some subset of the sample space. For example, an event could be rolling an even number on a six-sided die. This event corresponds to three outcomes, namely rolling a 2, 4, or 6 on the die.

To calculate probabilities, we can use several approaches, including relative frequency probability, which is based on actual data, and theoretical probability, which is based on theoretical conditions.

Relative Frequency Probability

Relative frequency probability is a method of determining the likelihood of an event occurring based on the observed frequency of its occurrence in a given sample or population. A data scientist conducts or observes a procedure and determines the number of times a certain Event $A$ occurs. The probability of Event $A$ , denoted as $P (A)$ , is then calculated based on data that has been collected from the experiment, as follows:

$P (A) = \frac{number of times Event A has occured}{number of times the procedure was repeated}$

Example 1

A polling organization asks a sample of 400 people if they are in favor of increased funding for local schools; 312 of the respondents indicate they are in favor of increased funding. Calculate the probability that a randomly selected person will be in favor of increased funding for local schools.

Using the data collected from this polling, a total of 400 people were asked the question, and 312 people were in favor of increased school funding. The probability for a randomly selected person being in favor of increased funding can then be calculated as follows (notice that Event $A$ in this example corresponds to the event that a person is in favor of the increased funding):

$\begin{array}{rcl} P (A) & = & Probability of In Favor \\ = & \frac{number of times Event A has occurred}{number of times the procedure was repeated} = \frac{312}{400} = 0.78 = 78 % \end{array}$

Example 2

A medical patient is told they need knee surgery, and they ask the doctor for an estimate of the probability of success for the surgical procedure. The doctor reviews data from the past two years and determines there were 200 such knee surgeries performed and 188 of them were successful. Based on this past data, the doctor calculates the probability of success for the knee surgery (notice that Event $A$ in this example corresponds to the event that a patient has a successful knee surgery result).

Using the data collected from the past two years, there were 200 surgeries performed, with 188 successes. The probability can then be calculated as:

$P (A) = Probability of Success = \frac{number of times Event A has occured}{number of times the procedure was repeated} = \frac{188}{200} = 0.94$

The doctor informs the patient that there is a 94% chance of success for the pending knee surgery.

Theoretical Probability

Theoretical probability is the method used when the outcomes in a probability experiment are equally likely—that is, under theoretical conditions.

The formula used for theoretical probability is similar to the formula used for empirical probability. Theoretical probability considers all the possible outcomes for an experiment that are known ahead of time so that past data is not needed in the calculation for theoretical probability.

$Theoretical Probability = \frac{number of outcomes for the event of interest}{total number of outcomes in the sample space}$

For example, the theoretical probability of rolling an even number when rolling a six-sided die is $\frac{3}{6}$ (which is $\frac{1}{2}$ , or 0.5). There are 3 outcomes corresponding to rolling an even number, and there are 6 outcomes total in the sample space. Notice this calculation can be done without conducting any experiments since the outcomes are equally likely.

Notice in Example 3 that probabilities can be written as fractions, decimals, or percentages.

Also note that any probability must be between 0 and 1 inclusive. An event with a probability of zero will never occur, and an event with a probability of 1 is certain to occur. A probability greater than 1 is not possible, and a negative probability is not possible.

Complement of an Event

The complement of an event is the set of all outcomes in the sample space that are not included in the event. The complement of Event $A$ is usually denoted by $A'$ (A prime). To find the probability of the complement of Event $A$ , subtract the probability of Event $A$ from 1.

$P (A') = 1 - P (A)$

Conditional Probability and Bayes’ Theorem

Data scientists are often interested in determining conditional probabilities, or the occurrence of one event that is conditional or dependent on another event. For example, a medical researcher might be interested to know if an asthma diagnosis for a patient is dependent on the patient’s exposure to air pollutants. In addition, when calculating conditional probabilities, we can sometimes revise a probability estimate based on additional information that is obtained. As we’ll see in the following section, Bayes’ Theorem allows new information to be used to refine a probability estimate.

Conditional Probability

A conditional probability is the probability of an event given that another event has already occurred. The notation for conditional probability is $P open parenthesis A | B close parenthesis$ , which denotes the probability of Event $A$ , given that Event $B$ has occurred. The vertical line between $A$ and $B$ denotes the “given” condition. (In this notation, the vertical line does not denote division).

For example, we might want to know the probability of a person getting a parking ticket given that a person did not put any money in a parking meter. Or a medical researcher might be interested in the probability of a patient developing heart disease given that the patient is a smoker.

If the occurrence of one event affects the probability of occurrence for another event, we say that the events are dependent; otherwise, the events are independent. Dependent events are events where the occurrence of one event affects the probability of occurrence of another event. Independent events are events where the probability of occurrence of one event is not affected by the occurrent of another event. The dependence of events has important implications in many fields such as marketing, engineering, psychology, and medicine.

Example 5

Determine if the two events are dependent or independent:

Rolling a 3 on one roll of a die, rolling a 4 on a second roll of a die
Obtaining heads on one flip of a coin and obtaining tails on a second flip of a coin
Selecting five basketball players from a professional basketball team and a player’s height is greater than 6 feet
Selecting an Ace from a deck of 52 cards, returning the card back to the original stack, and then selecting a King
Selecting an Ace from a deck of 52 cards, not returning the card back to the original stack, and then selecting a King

The result of one roll does not affect the result for the next roll, so these events are independent.
The results of one flip of the coin do not affect the results for any other flip of the coin, so these events are independent.
Typically, basketball players are tall individuals, and so they are more likely to have heights greater than 6 feet as opposed to the general public, so these events are dependent.
By selecting an Ace from a deck of 52 cards and then replacing the card, this restores the deck of cards to its original state, so the probability of selecting a King is not affected by the selection of the Ace. So these events are independent.
By selecting an Ace from a deck of 52 cards and then not replacing the card, this will result in only 51 cards remaining in the deck. Thus, the probability of selecting a King is affected by the selection of the Ace, so these events are dependent.

There are several ways to use conditional probabilities in data science applications.

Conditional probability can be defined as follows:

$P (A | B) = \frac{P (A and B)}{P (B)}, where P (B) \neq 0$

When assessing the conditional probability of $P open parenthesis A | B close parenthesis$ , if the two events are independent, this indicates that Event $A$ is not affected by the occurrence of Event $B$ , so we can write that $P open parenthesis A | B close parenthesis equals P open parenthesis A close parenthesis$ for independent events.

If we determine that the $P open parenthesis A | B close parenthesis$ is not equal to $P (A)$ , this indicates that the events are dependent.

$P open parenthesis A | B close parenthesis equals P open parenthesis A close parenthesis$ implies independent events, where $P open parenthesis B close parenthesis not equal to 0$ .

$P open parenthesis A | B close parenthesis not equal to P open parenthesis A close parenthesis$ implies dependent events.

Example 6

Table 3.2 shows the number of nursing degrees and non-nursing degrees at a university for a specific year, and the data is broken out by age groups. Calculate the probability that a randomly chosen graduate obtained a nursing degree, given that the graduate is in the age group of 23 and older.

**Table 3.2**
Age Group	Nursing Degrees	Non-Nursing Degrees	Total
22 and under	1036	1287	2323
23 and older	986	932	1918
Total	2022	2219	4241

Since we are given that the group of interest are those graduates in the age group of 23 and older, focus only on the second row in the table.

Looking only at the second row in the table, we are interested in the probability that a randomly chosen graduate obtained a nursing degree. The reduced sample space consists of 1,918 graduates, and 986 of them received nursing degrees. So the probability can be calculated as:

$P (nursing degree | age group of 23 and older) = \frac{986}{1,918} = 0.514$

Another method to analyze this example is to rewrite the conditional probability using the equation for $P (A and B)$ , as follows:

$\begin{array}{rcl} P (A and B) & = & P (A) \cdot P (B | A) \\ rewrite as: P (B | A) & = & \frac{P (A and B)}{P (A)} \end{array}$

We can now use this equation to calculate the probability that a randomly chosen graduate obtained a nursing degree, given that the graduate is in the age group of 23 and older. The probability of $A$ and $B$ is the probability that a graduate received a nursing degree and is also in the age group of 23 and older. From the table, there are 986 graduates who earned a nursing degree and are also in the age group of 23 and older. Since this number of graduates is out of the total sample size of 4,241, we can write the probability of Events $A$ and $B$ as:

$P (A and B) = \frac{986}{4,241}$

We can also calculate the probability that a graduate is in the age group of 23 and older. From the table, there are 1,918 graduates in this age group out of the total sample size of 4,241, so we can write the probability for Event $A$ as:

$P (A) = \frac{1,918}{4,241}$

Next, we can substitute these probabilities into the formula for $P open parenthesis B | A close parenthesis$ , as follows:

$P (B | A) = \frac{P (A and B)}{P (A)} = \frac{\frac{986}{4,241}}{\frac{1,918}{4,241}} = \frac{986}{4,241} \cdot \frac{4,241}{1,918} = \frac{986}{1,918} = 0.514$

Probability of At Least One

The probability of at least one occurrence of an event is often of interest in many data science applications. For example, a doctor might be interested to know the probability that at least one surgery to be performed this week will involve an infection of some type.

The phrase “at least one” implies the condition of one or more successes. From a sample space perspective, one or more successes is the complement of “no successes.” Using the complement rule discussed earlier, we can write the following probability formula:

$P (at least one success) = 1 - P (no successes)$

As an example, we can find the probability of rolling a die 3 times and obtaining at least one four on any of the rolls. This can be calculated by first finding the probability of not observing a four on any of the rolls and then subtracting this probability from 1. The probability of not observing a four on a roll of the die is 5/6. Thus, the probability of rolling a die 3 times and obtaining at least one four on any of the rolls is $1 - {(\frac{5}{6})}^{3} = 0.421$ .

Example 7

From past data, hospital administrators determine the probability that a knee surgery will be successful is 0.89.

During a certain day, the hospital schedules four knee surgeries to be performed. Calculate the probability that all four of these surgeries will be successful.
Calculate the probability that none of these knee surgeries will be successful.
Calculate the probability that at least one of the knee surgeries will be successful.

For all four surgeries to be successful, we can interpret that as the first surgery will be successful, and the second surgery will be successful, and the third surgery will be successful, and the fourth surgery will be successful. Since the probability of success for one knee surgery does not affect the probability of success for another knee surgery, we can assume these events are independent. Based on this, the probability that all four surgeries will be successful can be calculated using the probability formula for $P (A and B)$ by multiplying the probabilities together:

$\begin{array}{rcl} P (A and B and C and D) & = & P (A) \cdot P (B) \cdot P (C) \cdot P (D) \\ = & 0.89 \cdot 0.89 \cdot 0.89 \cdot 0.89 = 0.627 \end{array}$

There is about a 63% chance that all four knee surgeries will be successful.

The probability that a knee surgery will be unsuccessful can be calculated using the complement rule. If the probability of a successful surgery is 0.89, then the probability that the surgery will be unsuccessful is 0.11:

$P (A') = 1 - P (A) = 1 - 0.89 = 0.11$

Based on this, the probability that all four surgeries will be unsuccessful can be calculated using the probability formula for $P (A and B)$ by multiplying the probabilities together:

$\begin{array}{rcl} (A' and B' and C' and D') & = & P (A') \cdot P (B') \cdot P (C') \cdot P (D') \\ = & 0.11 \cdot 0.11 \cdot 0.11 \cdot 0.11 = 0.00015 \end{array}$

Since this is a very small probability, it is very unlikely that none of the surgeries will be successful.

To calculate the probability that at least one of the knee surgeries will be successful, use the probability formula for “at least one,” which is calculated as the complement of the event “none are successful.”

$\begin{array}{rcl} P (at least one success) & = & 1 - P (no successes) \\ P (at least one success) & = & 1 - 0.00015 = 0.99985 \end{array}$

This indicates there is a very high probability that at least one of the knee surgeries will be successful.

Bayes’ Theorem

Bayes’ Theorem is a statistical technique that allows for the revision of probability estimates based on new information or evidence that allows for more accurate and efficient decision-making in uncertain situations. Bayes’ Theorem is often used to help assess probabilities associated with medical diagnoses such as the probability a patient will develop cancer based on test screening results. This can be important in medical analysis to help assess the impact of a false positive, which is the scenario where the patient does not have the ailment but the screening test gives a false indication that the patient does have the ailment.

Bayes’ Theorem allows the calculation of the conditional probability $P open parenthesis A | B close parenthesis$ . There are several forms of Bayes’ Theorem, as shown:

$\begin{array}{rcl} P (A | B) & = & \frac{P (A) \cdot P (B | A)}{P (B)} \\ P (A | B) & = & \frac{P (A) \cdot P (B | A)}{P (A) \cdot P (B | A) + P (A^{'}) \cdot P (B | A^{'})} \end{array}$

Example 8

Assume that a certain type of cancer affects 3% of the population. Call the event that a person has cancer “Event $A$ ,” so:

$P open parenthesis A close parenthesis equals 0.03$

A patient can undergo a screening test for this type of cancer. Assume the probability of a true positive from the screening test is 75%, which indicates that probability that a person has a positive test result given that they actually have cancer is 0.75. Also assume the probability of a false positive from the screening test is 15%, which indicates that probability that a person has a positive test result given that they do not have cancer is 0.15.

A medical researcher is interested in calculating the probability that a patient actually has cancer given that the screening test shows a positive result.

The researcher is interested in calculating $P open parenthesis A | B close parenthesis$ , where Event $A$ is the person actually has cancer and Event $B$ is the event that the person shows a positive result in the screening test. Use Bayes’ Theorem to calculate this conditional probability.

From the example, the following probabilities are known:

$\begin{array}{rcl} P (A) & = & 0.03 \\ P (A') & = & 0.97 \end{array}$

The conditional probabilities can be interpreted as follows:

$\begin{array}{rcl} P (B | A) & = & P (positive test result | patient has cancer) = 0.75 \\ P (B | A') & = & P (positive test result | patient does not have cancer) = 0.15 \end{array}$

Substituting these probabilities into the formula for Bayes’ Theorem results in the following:

$P (A | B) = \frac{P (A) \cdot P (B | A)}{P (A) \cdot P (B | A) + P (A^{'}) \cdot P (B | A^{'})}$

$\begin{array}{rcl} P ( patient has cancer | positive test result) = \frac{0.03 \cdot 0.75}{0.03 \cdot 0.75 + 0.97 \cdot 0.15} = 0.134 \end{array}$

This result from Bayes’ Theorem indicates that even if a patient receives a positive test result from the screening test, this does not imply a high likelihood that the patient has cancer. There is only a 13% chance that the patient has cancer given a positive test result from the screening test.