11.5 Measures of Variation

Consider these three sets of quiz scores:

import math

# Four quiz sections. Every one has mean 5 and median 5 -- only the SPREAD differs.
# CHANGE any list (or add "Section E") and re-run.
sections = {
    "Section A": [5, 5, 5, 5, 5, 5, 5, 5, 5, 5],
    "Section B": [0, 0, 0, 0, 0, 10, 10, 10, 10, 10],
    "Section C": [4, 4, 4, 5, 5, 5, 5, 6, 6, 6],
    "Section D": [0, 5, 5, 5, 5, 5, 5, 5, 5, 10],
}

def std_dev(data, is_population=True):
    """The book's five steps, written out."""
    mean = sum(data) / len(data)
    squared = [(x - mean) ** 2 for x in data]           # steps 1 and 2
    total = sum(squared)                                # step 3
    divisor = len(data) if is_population else len(data) - 1   # step 4
    return math.sqrt(total / divisor), mean, total, divisor   # step 5

print("%-11s %7s %7s %10s %10s" % ("section", "mean", "range", "pop sd", "sample sd"))
print("-" * 50)
for name, data in sections.items():
    pop, mean, _, _ = std_dev(data, True)
    smp, _, _, _    = std_dev(data, False)
    print("%-11s %7.1f %7d %10.2f %10.2f" % (name, mean, max(data) - min(data), pop, smp))
print("-" * 50)
print("Sections B and D have the SAME range (10) but very different spread.")
print("Range only looks at two data values; standard deviation looks at all of them.")
print()

# The full worked table for one section -- change which one:
SHOW = "Section D"      # <-- CHANGE ME
data = sections[SHOW]
sd, mean, total, divisor = std_dev(data, True)
print("%s, step by step (mean = %.1f):" % (SHOW, mean))
print("%12s %26s %18s" % ("data value", "deviation: value - mean", "deviation squared"))
for x in data:
    print("%12d %26s %18.2f" % (x, "%d - %.1f = %+.1f" % (x, mean, x - mean), (x - mean) ** 2))
print("%12s %26s %18.2f" % ("", "sum of squared deviations", total))
print("divide by n = %d  ->  variance %.2f   (points SQUARED -- so take the root)"
      % (divisor, total / divisor))
print("population standard deviation = sqrt(%.2f) = %.2f points" % (total / divisor, sd))

Section A: 5 5 5 5 5 5 5 5 5 5
Section B: 0 0 0 0 0 10 10 10 10 10
Section C: 4 4 4 5 5 5 5 6 6 6

All three of these sets of data have a mean of 5 and median of 5, yet the sets of scores are clearly quite different. In section A, everyone had the same score; in section B half the class got no points and the other half got a perfect score, assuming this was a 10-point quiz. Section C was not as consistent as section A, but not as widely varied as section B.

In addition to the mean and median, which are measures of the "typical" or "middle" value, we also need a measure of how "spread out" or varied each data set is.

There are several ways to measure this "spread" of the data. The first is the simplest and is called the range.

In the last example, the range seems to be revealing how spread out the data is. However, suppose we add a fourth section, Section D, with scores 0 5 5 5 5 5 5 5 5 10.

This section also has a mean and median of 5. The range is 10, yet this data set is quite different than Section B. To better illuminate the differences, we’ll have to turn to more sophisticated measures of variation.

Using the data from section D, we could compute for each data value the difference between the data value and the mean:

data value	deviation: data value - mean
0	0-5=-5
5	5-5=0
5	5-5=0
5	5-5=0
5	5-5=0
5	5-5=0
5	5-5=0
5	5-5=0
5	5-5=0
10	10-5=5

We would like to get an idea of the "average" deviation from the mean, but if we find the average of the values in the second column the negative and positive values cancel each other out (this will always happen), so to prevent this we square every value in the second column:

We then add the squared deviations up to get $25 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 25 = 50$ . Ordinarily we would then divide by the number of scores, $n$ , (in this case, 10) to find the mean of the deviations. But we only do this if the data set represents a population; if the data set represents a sample (as it almost always does), we instead divide by $n minus 1$ (in this case, $10 - 1 = 9$ ).[1]

So in our example, we would have $\frac{50}{10} = 5$ if section D represents a population and $\frac{50}{9} =$ about 5.56 if section D represents a sample. These values (5 and 5.56) are called, respectively, the population variance and the sample variance for section D.

Variance can be a useful statistical concept, but note that the units of variance in this instance would be points-squared since we squared all of the deviations. What are points-squared? Good question. We would rather deal with the units we started with (points in this case), so to convert back we take the square root and get:

$population standard deviation = \sqrt{\frac{50}{10}} = \sqrt{5} \approx 2.2$

$sample standard deviation = \sqrt{\frac{50}{9}} \approx 2.4$

If we are unsure whether the data set is a sample or a population, we will usually assume it is a sample, and we will round answers to one more decimal place than the original data, as we have done above.

Example 2

Computing the standard deviation for Section B above, we first calculate that the mean is 5. Using a table can help keep track of your computations for the standard deviation:

data value	deviation: data value - mean	deviation squared
0	0-5=-5	$(- 5)^{2} = 25$
0	0-5=-5	$(- 5)^{2} = 25$
0	0-5=-5	$(- 5)^{2} = 25$
0	0-5=-5	$(- 5)^{2} = 25$
0	0-5=-5	$(- 5)^{2} = 25$
10	10-5=5	$open parenthesis 5 close parenthesis squared equals 25$
10	10-5=5	$open parenthesis 5 close parenthesis squared equals 25$
10	10-5=5	$open parenthesis 5 close parenthesis squared equals 25$
10	10-5=5	$open parenthesis 5 close parenthesis squared equals 25$
10	10-5=5	$open parenthesis 5 close parenthesis squared equals 25$

Assuming this data represents a population, we will add the squared deviations, divide by 10, the number of data values, and compute the square root:

$\sqrt{\frac{25 + 25 + 25 + 25 + 25 + 25 + 25 + 25 + 25 + 25}{10}} = \sqrt{\frac{250}{10}} = 5$

Notice that the standard deviation of this data set is much larger than that of section D since the data in this set is more spread out.

For comparison, the standard deviations of all four sections are:

Section A: 5 5 5 5 5 5 5 5 5 5	Standard deviation: 0
Section B: 0 0 0 0 0 10 10 10 10 10	Standard deviation: 5
Section C: 4 4 4 5 5 5 5 6 6 6	Standard deviation: 0.8
Section D: 0 5 5 5 5 5 5 5 5 10	Standard deviation: 2.2

Your Turn

Try it Now 7

The price of a jar of peanut butter at 5 stores were: $3.29, $3.59, $3.79, $3.75, and $3.99. Find the standard deviation of the prices.

Answer

Earlier we found the mean of the data was $3.682.

data value	deviation: data value - mean	deviation squared
3.29	3.29-3.682=-0.391	0.153664
3.59	3.59-3.682=-0.092	0.008464
3.79	3.79-3.682=0.108	0.011664
3.75	3.75-3.682=0.068	0.004624
3.99	3.99-3.682=0.308	0.094864

This data is from a sample, so we will add the squared deviations, divide by 4, the number of data values minus 1, and compute the square root:

$\sqrt{\frac{0.153664 + 0.008464 + 0.011664 + 0.004624 + 0.094864}{4}} \approx $ 0.261$

Where standard deviation is a measure of variation based on the mean, quartiles are based on the median.

While quartiles are not a 1-number summary of variation like standard deviation, the quartiles are used with the median, minimum, and maximum values to form a 5 number summary of the data.

To find the first quartile, we need to find the data value so that 25% of the data is below it. If $n$ is the number of data values, we compute a locator by finding 25% of $n$ . If this locator is a decimal value, we round up, and find the data value in that position. If the locator is a whole number, we find the mean of the data value in that position and the next data value. This is identical to the process we used to find the median, except we use 25% of the data values rather than half the data values as the locator.

Examples should help make this clearer.

Note that the median could be computed the same way, using 50%.

The 5-number summary combines the first and third quartile with the minimum, median, and maximum values.

Example 6

Returning to our quiz score data. In each case, the first quartile locator is $0.25 open parenthesis 10 close parenthesis equals 2.5$ , so the first quartile will be the 3^rd data value, and the third quartile will be the 8^th data value. Creating the five-number summaries:

Section and data	5-number summary
Section A: 5 5 5 5 5 5 5 5 5 5	5,5,5,5,5
Section B: 0 0 0 0 0 10 10 10 10 10	0,0,5,10,10
Section C: 4 4 4 5 5 5 5 6 6 6	4,4,5,6,6
Section D: 0 5 5 5 5 5 5 5 5 10	0,5,5,5,10

Of course, with a relatively small data set, finding a five-number summary is a bit silly, since the summary contains almost as many values as the original data.

Your Turn

Try it Now 8

The total cost of textbooks for the term was collected from 36 students. Find the 5 number summary of this data.

$140 $160 $160 $165 $180 $220 $235 $240 $250 $260 $280 $285

$285 $285 $290 $300 $300 $305 $310 $310 $315 $315 $320 $320

$330 $340 $345 $350 $355 $360 $360 $380 $395 $420 $460 $460

Answer

The data is already in order, so we don’t need to sort it first.

The minimum value is $140 and the maximum is $460.

There are 36 data values so $n = 36 . n / 2 = 18,$ which is a whole number, so the median is the mean of the $18 to the power th$ and $19 to the power th$ data values, $$ 305$ and $$ 310$ . The median is $$ 307.50$

To find the first quartile, we calculate the locator, $L = 0.25 (36) = 9 .$ since this is a whole number, we know $Q sub 1$ is the mean of the $9 to the power th$ and $10 to the power th$ data values, $$ 250$ and $$ 260. Q sub 1 equals$ $$ 255$

To find the third quartile, we calculate the locator, $L = 0.75 (36) = 27 .$ since this is a whole number, we know $Q sub 3$ is the mean of the $27 to the power th$ and $28 to the power th$ data values, $$ 345$ and $$ 350$ .
$Q sub 3 equals $ 347.50$

The 5 number summary of this data is: $140, $255, $307.50, $347.50, $460

Example 7

Returning to the household income data from earlier, create the five-number summary.

Income (thousands of dollars)	Frequency
15	6
20	8
25	11
30	17
35	19
40	20
45	12
50	7

Show solution

By adding the frequencies, we can see there are 100 data values represented in the table. In Example 20, we found the median was $35 thousand. We can see in the table that the minimum income is $15 thousand, and the maximum is $50 thousand.

To find $Q sub 1$ , we calculate the locator: L = 0.25(100) = 25. This is a whole number, so $Q sub 1$ will be the mean of the 25^th and 26^th data values.

Counting up in the data as we did before,

$\begin{matrix} There are 6 data values of $15, so & Values 1 to 6 are $15 thousand \\ The next 8 data values are $20, so & Values 7 to (6+8)=14 are $20 thousand \\ The next 11 data values are $25, so & Values 15 to (14+11)=25 are $25 thousand \\ The next 17 data values are $30, so & Values 26 to (25+17)=42 are $30 thousand \end{matrix}$

The 25^th data value is $25 thousand, and the 26^th data value is $30 thousand, so $Q sub 1$ will be the mean of these: $(25 + 30) / 2 = $ 27.5$ thousand.

To find $Q sub 3$ , we calculate the locator: $L = 0.75 (100) = 75$ . This is a whole number, so $Q sub 3$ will be the mean of the 75^th and 76^th data values. Continuing our counting from earlier,

$\begin{matrix} The next 19 data values are 35, s o & Values 43 to (42+19)=61 are $35 thousand \\ The next 20 data values are $40, so & Values 61 to (61+20)=81 are $40 thousand \end{matrix}$

Both the 75^th and 76^th data values lie in this group, so $Q sub 3$ will be $40 thousand.

Putting these values together into a five-number summary, we get: 15, 27.5, 35, 40, 50

Note that the 5 number summary divides the data into four intervals, each of which will contain about 25% of the data. In the previous example, that means about 25% of households have income between $40 thousand and $50 thousand.

For visualizing data, there is a graphical representation of a 5-number summary called a box plot, or box and whisker graph.

To create a box plot, a number line is first drawn. A box is drawn from the first quartile to the third quartile, and a line is drawn through the box at the median. “Whiskers” are extended out to the minimum and maximum values.

Box plots are particularly useful for comparing data from two populations.

import math
import matplotlib.pyplot as plt

def locate(sorted_data, fraction):
    """The book's locator rule: L = fraction * n; round up, or average two values."""
    n = len(sorted_data)
    L = fraction * n
    if L == int(L):                       # whole number -> mean of the Lth and (L+1)th
        L = int(L)
        return (sorted_data[L - 1] + sorted_data[L]) / 2
    return sorted_data[math.ceil(L) - 1]  # decimal -> round up, take that value

def five_number(data):
    s = sorted(data)
    return [s[0], locate(s, 0.25), locate(s, 0.50), locate(s, 0.75), s[-1]]

# Book data sets. CHANGE any list, or paste your own, and re-run.
sets = {
    "9 female heights (in)": [59, 60, 62, 64, 66, 67, 69, 70, 72],
    "8 female heights (in)": [59, 60, 62, 64, 66, 67, 69, 70],
    "36 textbook costs ($)": [140, 160, 160, 165, 180, 220, 235, 240, 250, 260, 280,
                              285, 285, 285, 290, 300, 300, 305, 310, 310, 315, 315,
                              320, 320, 330, 340, 345, 350, 355, 360, 360, 380, 395,
                              420, 460, 460],
}

print("%-24s %8s %8s %8s %8s %8s %9s" % ("data set", "min", "Q1", "median", "Q3", "max", "IQR"))
print("-" * 76)
for name, data in sets.items():
    mn, q1, med, q3, mx = five_number(data)
    print("%-24s %8.2f %8.2f %8.2f %8.2f %8.2f %9.2f" % (name, mn, q1, med, q3, mx, q3 - q1))
print("-" * 76)
print("Each of the four pieces holds about 25% of the data -- so a WIDE box means")
print("the middle half of the data is spread out, not that there are more values in it.")

# Boxplot of the two height samples, drawn straight from the book's five-number summary.
fig, ax = plt.subplots(figsize=(6, 3.4))
stats = []
for name in ("9 female heights (in)", "8 female heights (in)"):
    mn, q1, med, q3, mx = five_number(sets[name])
    stats.append({"label": name, "med": med, "q1": q1, "q3": q3,
                  "whislo": mn, "whishi": mx, "fliers": []})
ax.bxp(stats, showfliers=False, patch_artist=True,
       boxprops={"facecolor": "#dbe5f1"}, medianprops={"color": "#c00000"})
ax.set_ylabel("height (inches)")
ax.set_title("Five-number summaries as box plots")
plt.tight_layout()
plt.show()

Example 11

The boxplot below is based on the birth weights of infants with severe idiopathic respiratory distress syndrome (SIRDS)[2]. The boxplot is separated to show the birth weights of infants who survived and those that did not.

Comparing the two groups, the boxplot reveals that the birth weights of the infants that died appear to be, overall, smaller than the weights of infants that survived. In fact, we can see that the median birth weight of infants that survived is the same as the third quartile of the infants that died.

Similarly, we can see that the first quartile of the survivors is larger than the median weight of those that died, meaning that over 75% of the survivors had a birth weight larger than the median birth weight of those that died.

Looking at the maximum value for those that died and the third quartile of the survivors, we can see that over 25% of the survivors had birth weights higher than the heaviest infant that died.

The box plot gives us a quick, albeit informal, way to determine that birth weight is quite likely linked to survival of infants with SIRDS.

[1] The reason we do this is highly technical, but we can see how it might be useful by considering the case of a small sample from a population that contains an outlier, which would increase the average deviation: the outlier very likely won't be included in the sample, so the mean deviation of the sample would underestimate the mean deviation of the population; thus we divide by a slightly smaller number to get a slightly bigger average deviation.

[2] van Vliet, P.K. and Gupta, J.M. (1973) Sodium bicarbonate in idiopathic respiratory distress syndrome. Arch. Disease in Childhood, 48, 249–255. As quoted on http://openlearn.open.ac.uk/mod/ouco...&section=1.1.3

Adapted from Math in Society by David Lippman, hosted on LibreTexts (math.libretexts.org) and licensed under CC BY-SA 3.0. Changes were made. License: CC-BY-SA-3.0.