#set document(title: "3.2 Measures of Variation", author: "OpenStax / XYZ Homework") #set page(width: 8.5in, height: auto, margin: 1in) #import "@preview/cetz:0.5.2" #set text(font: ("STIX Two Text", "Libertinus Serif", "New Computer Modern"), size: 10.5pt, lang: "en") #show math.equation: set text(font: ("STIX Two Math", "New Computer Modern Math")) #set par(justify: true, leading: 0.62em, spacing: 0.9em) #set enum(spacing: 1.1em) // room between list items so tall inline fractions don't collide #set list(spacing: 1.1em) #set table(stroke: 0.5pt + rgb("#c7ccd3")) #let BLUE = rgb("#183B6F") // brand navy — section bars + example/solution labels (white on navy 11.09:1) #let ORANGE = rgb("#A94509") // brand primary-700 — AA-safe deep orange for TEXT (5.93:1 on white; raw brand #F37021 is 2.94:1 and must never carry text) #let RED = rgb("#DC2626") // brand error-600 #let GREEN = rgb("#059669") // brand success-600 (decoration only; small green text uses green-text #007942) #show heading.where(level: 1): it => block(width: 100%, above: 0pt, below: 16pt, fill: gradient.linear(BLUE, rgb("#2C5AA0")), inset: (x: 14pt, y: 12pt), radius: 3pt, text(fill: white, weight: "bold", size: 19pt, it.body)) #show heading.where(level: 2): it => block(width: 100%, above: 18pt, below: 10pt, fill: BLUE, inset: (x: 10pt, y: 6pt), radius: 2pt, text(fill: white, weight: "bold", size: 12pt, it.body)) #show heading.where(level: 3): it => text(fill: ORANGE, weight: "bold", size: 12.5pt, it.body) #show heading.where(level: 4): it => text(fill: BLUE, weight: "bold", size: 10.5pt, it.body) #let examplebox(label, title, body) = block(width: 100%, breakable: true, fill: rgb("#EFF1F5"), stroke: 0.5pt + rgb("#CFDDF0"), radius: 4pt, inset: 10pt, above: 12pt, below: 12pt)[ #block(below: 6pt)[#box(fill: BLUE, inset: (x: 6pt, y: 2pt), radius: 2pt, text(fill: white, weight: "bold", size: 8.5pt, label)) #h(0.4em) #strong[#title]] #body] // rail = decorative left rule (raw brand token); labelcolor = AA-safe label text shade #let notebox(label, rail, labelcolor, tint, body) = block(width: 100%, breakable: true, fill: tint, stroke: (left: 3pt + rail), inset: (left: 10pt, rest: 8pt), radius: (right: 4pt), above: 11pt, below: 11pt)[ #text(fill: labelcolor, weight: "bold", size: 7.5pt, tracking: 0.5pt)[#upper(label)] #linebreak() #body] #let solutionbox(body) = block(above: 4pt, below: 8pt)[ #text(fill: BLUE, weight: "bold", size: 8.5pt)[Solution] #linebreak() #body] #let figph(msg) = block(width: 100%, height: 60pt, fill: rgb("#f6f7f9"), stroke: (paint: rgb("#c7ccd3"), dash: "dashed"), radius: 4pt, inset: 10pt)[ #align(center + horizon, text(fill: rgb("#889"), style: "italic", size: 9pt, msg))] // Standardize inlined figure sizes: measure the natural CeTZ canvas, then scale to a // consistent envelope (aspect-aware; see build_typst.py FIG_* constants). Unlike the // print preamble, dimensions are FLOORED: in an editor a user can trim a figure to a // degenerate 1-D shape (a bare line), and w/h or tw/w would then divide by zero. #let _STD_W = 3.5 #let _WIDE_W = 5.6 #let _MAX_H = 3.4 #let _ASPECT_WIDE = 2.2 #let _UPSCALE_MAX = 1.15 #let stdfig(body) = context { let m = measure(body) let w = calc.max(m.width / 1in, 0.01) let h = calc.max(m.height / 1in, 0.01) let tw = if w / h > _ASPECT_WIDE { _WIDE_W } else { _STD_W } let s = calc.min(tw / w, _MAX_H / h, _UPSCALE_MAX) align(center, box(scale(x: s * 100%, y: s * 100%, reflow: true, body))) } #show figure: set block(breakable: false) #set figure(gap: 8pt) #show figure.caption: set text(size: 8.5pt, fill: rgb("#555")) == 3.2#h(0.6em)Measures of Variation === Learning Outcomes By the end of this section, you should be able to: - Define and calculate the range, the variance, and the standard deviation for a dataset. - Use Python to calculate measures of variation for a dataset. Providing some measure of the spread, or #emph[variation], in a dataset is crucial to a comprehensive summary of the dataset. Two datasets may have the same mean but can exhibit very different spread, and so a measure of dispersion for a dataset is very important. While measures of central tendency (like mean, median, and mode) describe the center or average value of a distribution, measures of dispersion give insights into how much individual data points deviate from this central value. The following two datasets are the exam scores for a group of three students in a biology course and in a statistics course. Dataset A: Exam scores for students in a biology course: 40, 70, 100 #linebreak() Dataset B: Exam scores for students in a statistics course: 69, 70, 71 Notice that the mean score for both Dataset A and Dataset B is 70. However, the datasets are significantly different from one another: Dataset A has larger variability where one student scored 30 points below the mean and another student scored 30 points above the mean. #linebreak() Dataset B has smaller variability where the exam scores are much more tightly clustered around the mean of 70. This example illustrates that publishing the mean of a dataset is often inadequate to fully communicate the characteristics of the dataset. Instead, data scientists will typically include a measure of variation as well. The three primary measures of variability are range, variance, and standard deviation, and these are described next. === Range #strong[Range] is a measure of dispersion for a dataset that is calculated by subtracting the minimum from the maximum of the dataset: #math.equation(block: true, alt: "Range equals Max minus Min")[$"Range" #h(0.2em) = "Max" #h(0.2em) − "Min"$] Range is a straightforward calculation but makes use of only two of the data values in a dataset. The range can also be affected by outliers. #examplebox("Example 1")[][ Calculate the range for Dataset A and Dataset B: Dataset A: Exam scores for students in a biology course: 40, 70, 100 #linebreak() Dataset B: Exam scores for students in a statistics course: 69, 70, 71 #solutionbox[ For Dataset A, the maximum data value is 100 and the minimum data value is 40. #linebreak() The range is then calculated as: #math.equation(block: true, alt: "Range, equals, Max minus Min; Range, equals, 100 minus 40; Range, equals, 60")[$"Range" & = & "Max" #h(0.2em) − "Min" \ "Range" & = & 100 − 40 \ "Range" & = & 60$] For Dataset B, the maximum data value is 71 and the minimum data value is 69. #linebreak() The range is then calculated as: #math.equation(block: true, alt: "Range, equals, Max minus Min; Range, equals, 71 minus 69; Range, equals, 2")[$"Range" & = & "Max" #h(0.2em) − "Min" \ "Range" & = & 71 − 69 \ "Range" & = & 2$] The range clearly indicates that there is much less spread in Dataset B as compared to Dataset A. ] ] One drawback to the use of the range is that it doesn’t take into account every data value. The range only uses two data values from the dataset: the minimum (min) and the maximum (max). Also the range is influenced by outliers since an outlier might appear as a minimum or maximum data value and thus skew the results. For these reasons, we typically use other measures of variation, such as variance or standard deviation. === Variance The #strong[variance] provides a measure of the spread of data values by using the squared deviations from the mean. The more the individual data values differ from the mean, the larger the variance. A financial advisor might use variance to determine the volatility of an investment and therefore help guide financial decisions. For example, a more cautious investor might opt for investments with low volatility. The formula used to calculate variance also depends on whether the data is collected from a sample or a population. The notation #math.equation(block: false, alt: "s squared")[$s^(2)$] is used to represent the #emph[#strong[sample variance]], and the notation #math.equation(block: false, alt: "σ squared")[$σ^(2)$] is used to represent the #emph[#strong[population variance]]. Formula for the sample variance: #math.equation(block: true, alt: "s squared equals the fraction ∑ open parenthesis x minus x minus close parenthesis squared over n minus 1")[$s^(2) = frac(∑ ( x − limits(x)^(−) )^(2), n − 1)$] Formula for the population variance: #math.equation(block: true, alt: "σ squared equals the fraction ∑ open parenthesis x minus µ close parenthesis squared over N")[$σ^(2) = frac(∑ ( x − µ )^(2), N)$] In these formulas: #linebreak() #math.equation(block: false, alt: "x")[$x$] represents the individual data values #linebreak() #math.equation(block: false, alt: "x minus")[$limits(x)^(−)$] represents the sample mean #linebreak() #math.equation(block: false, alt: "n")[$n$] represents the sample size #linebreak() #math.equation(block: false, alt: "µ")[$µ$] represents the population mean #linebreak() #math.equation(block: false, alt: "N")[$N$] represents the population size #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ #emph[Alternate Formula for Variance] An alternate formula for the variance is available. It is sometimes used for more efficient computations: #math.equation(block: true, alt: "σ squared equals the fraction ∑ x squared over N minus µ squared")[$σ^(2) = frac(∑ x^(2), N) − µ^(2)$] ] In the formulas for sample variance and population variance, notice the denominator for the sample variance is #math.equation(block: false, alt: "n minus 1")[$n − 1$], whereas the denominator for the population variance is #math.equation(block: false, alt: "N")[$N$]. The use of #math.equation(block: false, alt: "n minus 1")[$n − 1$] in the denominator of the sample variance is used to provide the best estimate for the population variance, in the sense that if repeated samples of size #math.equation(block: false, alt: "n")[$n$] are taken and the sample mean computed each time, then the average of those sample means will tend to the population mean as the number of repeated samples increase. It is important to note that in many data science applications, population data is unavailable, and so we typically calculate the sample variance. For example, if a researcher wanted to estimate the percentage of smokers for all adults in the United States, it would be impractical to collect data from every adult in the United States. Notice that the sample variance is a #strong[sum of squares]. Its units of measurement are squares of the units of measurement of the original data. Since these square units are different than the units in the original data, this can be confusing. By contrast, standard deviation is measured in the same units as the original dataset, and thus the standard deviation is more commonly used to measure the spread of a dataset. === Standard Deviation The #strong[standard deviation] of a dataset provides a numerical measure of the overall amount of variation #emph[in a dataset in the same units as the data]; it can be used to determine whether a particular data value is close to or far from the mean, relative to the typical distance from the mean. The standard deviation is always positive or zero. It is small when the data values are all concentrated close to the mean, exhibiting little variation, or spread. It is larger when the data values are spread out more from the mean, exhibiting more variation. A smaller standard deviation implies less variability in a dataset, and a larger standard deviation implies more variability in a dataset. Suppose that we are studying the variability of two companies (A and B) with respect to employee salaries. The average salary for both companies is \$60,000. For Company A, the standard deviation of salaries is \$8,000, whereas the standard deviation for salaries for Company B is \$19,000. Because Company B has a higher standard deviation, we know that there is more variation in the employee salaries for Company B as compared to Company A. There are two different formulas for calculating standard deviation. Which formula to use depends on whether the data represents a sample or a population. The notation #math.equation(block: false, alt: "s")[$s$] is used to represent the sample standard deviation, and the notation #math.equation(block: false, alt: "σ")[$σ$] is used to represent the population standard deviation. In the formulas shown, #math.equation(block: false, alt: "x minus")[$limits(x)^(−)$] is the sample mean, #math.equation(block: false, alt: "µ")[$µ$] is the population mean, #math.equation(block: false, alt: "n")[$n$] is the sample size, and #math.equation(block: false, alt: "N")[$N$] is the population size. #linebreak() Formula for the sample standard deviation: #math.equation(block: true, alt: "s equals the square root of the fraction ∑ open parenthesis x minus x minus close parenthesis squared over n minus 1")[$s = sqrt(frac(∑ ( x − limits(x)^(−) )^(2), n − 1))$] Formula for the population standard deviation: #math.equation(block: true, alt: "σ equals the square root of the fraction ∑ open parenthesis x minus µ close parenthesis squared over N")[$σ = sqrt(frac(∑ ( x − µ )^(2), N))$] Notice that the sample standard deviation is calculated as the square root of the variance. This means that once the sample variance has been calculated, the sample standard deviation can then be easily calculated as the square root of the sample variance, as in Example 2. #examplebox("Example 2")[][ A biologist calculates that the sample variance for the amount of plant growth for a sample of plants is 8.7 cm#super[2]. Calculate the sample standard deviation. #solutionbox[ The sample standard deviation (#math.equation(block: false, alt: "s")[$s$]) is calculated as the square root of the variance. #math.equation(block: true, alt: "s equals the square root of s squared equals the square root of 8.7 equals 2.9 cm")[$s = sqrt(s^(2)) = sqrt(8.7) = 2.9 " cm"$] ] ] #examplebox("Example 3")[][ Assume the sample variance (#math.equation(block: false, alt: "s squared")[$s^(2)$]) for a dataset is calculated as 42.2. Based on this, calculate the sample standard deviation. #solutionbox[ The sample standard deviation (#math.equation(block: false, alt: "s")[$s$]) is calculated as the square root of the variance. #math.equation(block: true, alt: "s equals the square root of s squared equals the square root of 42.2 equals 6.5 years")[$s = sqrt(s^(2)) = sqrt(42.2) = 6.5 " years"$] This result indicates that the standard deviation is about 6.5 years. ] ] Notice that the sample variance is the square of the sample standard deviation, so if the sample standard deviation is known, the sample variance can easily be calculated. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ #emph[Use of Technology for Calculating Measures of Variability] Due to the complexity of calculating variance and standard deviation, technology is typically utilized to calculate these measures of variability. For example, refer to the examples shown in Coefficient of Variation on using Python for measures of variation. ] === Coefficient of Variation A data scientist might be interested in comparing variation with different units of measurement of different means, and in these scenarios the #strong[coefficient of variation (CV)] can be used. The coefficient of variation measures the variation of a dataset by calculating the standard deviation as a percentage of the mean. Note: coefficient of variation is typically expressed in a percentage format. #math.equation(block: true, alt: "CV, equals, the fraction σ over μ times 100 %; Sample CV, equals, the fraction s over x minus times 100 %")[$"CV" & = & frac(σ, μ) × 100 % \ "Sample CV" & = & frac(s, limits(x)^(−)) × 100 %$] #examplebox("Example 4")[][ Compare the relative variability for Company A versus Company B using the coefficient of variation, based on the following sample data: Company A: #math.equation(block: false, alt: "Sample Mean equals $ 68,000 , Sample Standard Deviation equals $ 9,200")[$"Sample Mean" #h(0.2em) = \$ 68,000 , " Sample Standard Deviation" #h(0.2em) = \$ 9,200$] Company B: #math.equation(block: false, alt: "Sample Mean equals $ 71,000 , Sample Standard Deviation equals $ 6,400")[$"Sample Mean" #h(0.2em) = \$ 71,000 , " Sample Standard Deviation" #h(0.2em) = \$ 6,400$] #solutionbox[ Calculate the coefficient of variation for each company: #math.equation(block: true, alt: "CV for Company A equals the fraction s over x minus times 100 % equals the fraction 9,200 over 68,000 times 100 % equals 13.5 %; CV for Company B equals the fraction s over x minus times 100 % equals the fraction 6,400 over 71,000 times 100 % equals 9.0 %")[$"CV for Company A" = frac(s, limits(x)^(−)) × 100 % = frac(9","200, 68","000) × 100 % = 13.5 % \ "CV for Company B" = frac(s, limits(x)^(−)) × 100 % = frac(6","400, 71","000) × 100 % = 9.0 %$] Company A exhibits more variability relative to the mean as compared to Company B. ] ] === Using Python for Measures of Variation #strong[DataFrame.describe()] computes standard deviation as well on each column of a dataset. The #strong[std] lists the standard deviation of each column. #figure(figph[A data table summarizing statistics about 966 items in the “movie profit” dataset, with columns for “unnamed: 0,” “rating,” “duration,” “US gross” and “worldwide gross.” The standard deviation row is highlighted. The standard deviation is about 0.89 for ratings and about 21.6 for durations. The standard deviation for US gross earnings is about \$110.6 million and for worldwide gross earnings about \$294.76 million.], alt: "A data table summarizing statistics about 966 items in the “movie profit” dataset, with columns for “unnamed: 0,” “rating,” “duration,” “US gross” and “worldwide gross.” The standard deviation row is highlighted. The standard deviation is about 0.89 for ratings and about 21.6 for durations. The standard deviation for US gross earnings is about $110.6 million and for worldwide gross earnings about $294.76 million.", caption: [The Output of DataFrame.describe() with the Movie Profit Dataset])