#set document(title: "12.5 Pairwise Comparisons", author: "OpenStax") #set page(width: 8.5in, height: auto, margin: 1in) #import "@preview/cetz:0.5.2" #set text(font: ("STIX Two Text", "Libertinus Serif", "New Computer Modern"), size: 10.5pt, lang: "en") #show math.equation: set text(font: ("STIX Two Math", "New Computer Modern Math")) #set par(justify: true, leading: 0.62em, spacing: 0.9em) #set enum(spacing: 1.1em) // room between list items so tall inline fractions don't collide #set list(spacing: 1.1em) #set table(stroke: 0.5pt + rgb("#c7ccd3")) #let BLUE = rgb("#183B6F") // brand navy — section bars + example/solution labels (white on navy 11.09:1) #let ORANGE = rgb("#A94509") // brand primary-700 — AA-safe deep orange for TEXT (5.93:1 on white; raw brand #F37021 is 2.94:1 and must never carry text) #let RED = rgb("#DC2626") // brand error-600 #let GREEN = rgb("#059669") // brand success-600 (decoration only; small green text uses green-text #007942) #show heading.where(level: 1): it => block(width: 100%, above: 0pt, below: 16pt, fill: gradient.linear(BLUE, rgb("#2C5AA0")), inset: (x: 14pt, y: 12pt), radius: 3pt, text(fill: white, weight: "bold", size: 19pt, it.body)) #show heading.where(level: 2): it => block(width: 100%, above: 18pt, below: 10pt, fill: BLUE, inset: (x: 10pt, y: 6pt), radius: 2pt, text(fill: white, weight: "bold", size: 12pt, it.body)) #show heading.where(level: 3): it => text(fill: ORANGE, weight: "bold", size: 12.5pt, it.body) #show heading.where(level: 4): it => text(fill: BLUE, weight: "bold", size: 10.5pt, it.body) #let examplebox(label, title, body) = block(width: 100%, breakable: true, fill: rgb("#EFF1F5"), stroke: 0.5pt + rgb("#CFDDF0"), radius: 4pt, inset: 10pt, above: 12pt, below: 12pt)[ #block(below: 6pt)[#box(fill: BLUE, inset: (x: 6pt, y: 2pt), radius: 2pt, text(fill: white, weight: "bold", size: 8.5pt, label)) #h(0.4em) #strong[#title]] #body] // rail = decorative left rule (raw brand token); labelcolor = AA-safe label text shade #let notebox(label, rail, labelcolor, tint, body) = block(width: 100%, breakable: true, fill: tint, stroke: (left: 3pt + rail), inset: (left: 10pt, rest: 8pt), radius: (right: 4pt), above: 11pt, below: 11pt)[ #text(fill: labelcolor, weight: "bold", size: 7.5pt, tracking: 0.5pt)[#upper(label)] #linebreak() #body] #let solutionbox(body) = block(above: 4pt, below: 8pt)[ #text(fill: BLUE, weight: "bold", size: 8.5pt)[Solution] #linebreak() #body] #let figph(msg) = block(width: 100%, height: 60pt, fill: rgb("#f6f7f9"), stroke: (paint: rgb("#c7ccd3"), dash: "dashed"), radius: 4pt, inset: 10pt)[ #align(center + horizon, text(fill: rgb("#889"), style: "italic", size: 9pt, msg))] // Standardize inlined figure sizes: measure the natural CeTZ canvas, then scale to a // consistent envelope (aspect-aware; see build_typst.py FIG_* constants). Unlike the // print preamble, dimensions are FLOORED: in an editor a user can trim a figure to a // degenerate 1-D shape (a bare line), and w/h or tw/w would then divide by zero. #let _STD_W = 3.5 #let _WIDE_W = 5.6 #let _MAX_H = 3.4 #let _ASPECT_WIDE = 2.2 #let _UPSCALE_MAX = 1.15 #let stdfig(body) = context { let m = measure(body) let w = calc.max(m.width / 1in, 0.01) let h = calc.max(m.height / 1in, 0.01) let tw = if w / h > _ASPECT_WIDE { _WIDE_W } else { _STD_W } let s = calc.min(tw / w, _MAX_H / h, _UPSCALE_MAX) align(center, box(scale(x: s * 100%, y: s * 100%, reflow: true, body))) } #show figure: set block(breakable: false) #set figure(gap: 8pt) #show figure.caption: set text(size: 8.5pt, fill: rgb("#555")) == 12.5#h(0.6em)Pairwise Comparisons #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ #emph[Prerequisites] Difference Between Two Means (Independent Groups) #linebreak() #linebreak() ] #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ #emph[Learning Objectives] + Define pairwise comparison + Describe the problem with doing t tests among all pairs of means + Calculate the Tukey HSD test + Explain why the Tukey test should not necessarily be considered a follow-up test ] Many experiments are designed to compare more than two conditions. We will take as an example the case study #link("https://onlinestatbook.com/2/case_studies/leniency.html")["Smiles and Leniency]." In this study, the effect of different types of smiles on the leniency shown to a person was investigated. An obvious way to proceed would be to do a t test of the difference between each group mean and each of the other group means. This procedure would lead to the six comparisons shown in Table 1. Table 1. Six Comparisons among Means. #figure(table( columns: 3, align: left, inset: 6pt, table.header([false vs felt], [], []), [false vs miserable], [], [], [false vs neutral], [], [], [felt vs miserable], [], [], [felt vs neutral], [], [], [miserable vs neutral], [], [], )) The problem with this approach is that if you did this analysis, you would have six chances to make a Type I error. Therefore, if you were using the 0.05 significance level, the probability that you would make a Type I error on at least one of these comparisons is greater than 0.05. The more means that are compared, the more the Type I error rate is inflated. Figure 1 shows the number of possible comparisons between pairs of means (#strong[pairwise comparisons]) as a function of the number of means. If there are only two means, then only one comparison can be made. If there are 12 means, then there are 66 possible comparisons. #linebreak() #figure(figph[Line graph of the Number of Comparisons (y-axis 0 to 70) against the Number of Means (x-axis 2 to 12). The curve accelerates upward from 1 comparison for 2 means through 10 at 5 means and 28 at 8 means to 66 at 12 means — the pairwise comparisons grow as k(k-1)/2.], alt: "Line graph of the Number of Comparisons (y-axis 0 to 70) against the Number of Means (x-axis 2 to 12). The curve accelerates upward from 1 comparison for 2 means through 10 at 5 means and 28 at 8 means to 66 at 12 means — the pairwise comparisons grow as k(k-1)/2.", caption: [Figure 1. Number of pairwise comparisons as a function of the number of means.]) Figure 2 shows the probability of a Type I error as a function of the number of means. As you can see, if you have an experiment with 12 means, the probability is about 0.70 that at least one of the 66 comparisons among means would be significant even if all 12 population means were the same. #linebreak() #figure(figph[Line graph of the Probability of a Type I Error (y-axis 0 to 0.8) against the Number of Means (x-axis 2 to 12). The curve rises from 0.05 at 2 means, passes about 0.30 at 5 means and 0.60 at 9, and flattens toward about 0.70 at 12 — with 12 means the chance that at least one of the 66 comparisons is significant is about 0.70 even when every population mean is equal.], alt: "Line graph of the Probability of a Type I Error (y-axis 0 to 0.8) against the Number of Means (x-axis 2 to 12). The curve rises from 0.05 at 2 means, passes about 0.30 at 5 means and 0.60 at 9, and flattens toward about 0.70 at 12 — with 12 means the chance that at least one of the 66 comparisons is significant is about 0.70 even when every population mean is equal.", caption: [Figure 2. Probability of a Type I error as a function of the number of means.]) The Type I error rate can be controlled using a test called the Tukey Honestly Significant Difference test or #strong[Tukey HSD] for short. The Tukey HSD is based on a variation of the#strong[t distribution] that takes into account the number of means being compared. This distribution is called the #strong[studentized range distribution]. Let's return to the leniency study to see how to compute the Tukey HSD test. You will see that the computations are very similar to those of an independent-groups t test. The steps are outlined below: + Compute the means and variances of each group. They are shown below. #linebreak() #linebreak() #linebreak() #figure(table( columns: 3, align: left, inset: 6pt, table.header([Condition], [Mean], [Variance]), [False], [5.37], [3.34], [Felt], [4.91], [2.83], [Miserable], [4.91], [2.11], [Neutral], [4.12], [2.32], )) + Compute MSE, which is simply the mean of the variances. It is equal to 2.65. #linebreak() #linebreak() + Compute #linebreak() #linebreak() #math.equation(block: true, alt: "Q equals the fraction M sub i minus M sub j over the square root of MSE / n")[$Q = frac(M_(i) − M_(j), sqrt("MSE" / n))$] #linebreak() #linebreak() for each pair of means, where M#sub[i] is one mean, M#sub[j] is the other mean, and n is the number of scores in each group. For these data, there are 34 observations per group. The value in the denominator is 0.2791. #linebreak() #linebreak() + Compute p for each comparison using the #link("https://onlinestatbook.com/2/calculators/studentized_range_dist.html")[Studentized Range Calculator]. The degrees of freedom is equal to the total number of observations minus the number of means. For this experiment, df = 136 - 4 = 132. #linebreak() #link("https://onlinestatbook.com/2/calculators/studentized_range_dist.html")[Studentized Range Calculator] The tests for these data are shown in Table 2. Table 2. Six Pairwise Comparisons. #figure(table( columns: 4, align: left, inset: 6pt, table.header([Comparison], [M#sub[i]-M#sub[j]], [Q], [p]), [False - Felt], [0.456], [1.633], [0.656], [False - Miserable], [0.456], [1.633], [0.656], [False - Neutral], [1.25], [4.478], [0.010], [Felt - Miserable], [0.00], [0.00], [1.000], [Felt - Neutral], [0.794], [2.845], [0.189], [Miserable - Neutral], [0.794], [2.845], [0.189], )) The only significant comparison is between the false smile and the neutral smile. It is not unusual to obtain results that on the surface appear paradoxical. For example, these results appear to indicate that (a) the false smile is the same as the miserable smile, (b) the miserable smile is the same as the neutral control, and (c) the false smile is different from the neutral control. This apparent contradiction is avoided if you are careful not to accept the null hypothesis when you fail to reject it. The finding that the false smile is not significantly different from the miserable smile does not mean that they are really the same. Rather it means that there is not convincing evidence that they are different. Similarly, the non-significant difference between the miserable smile and the control does not mean that they are the same. The proper conclusion is that the false smile is higher than the control and that the miserable smile is either (a) equal to the false smile, (b) equal to the control, or (c) somewhere in-between. The assumptions of the Tukey test are essentially the same as for an independent-groups t test: normality, homogeneity of variance, and independent observations. The test is quite robust to violations of normality. Violating homogeneity of variance can be more problematical than in the two-sample case since the MSE is based on data from all groups. The assumption of independence of observations is important and should not be violated. === Computer Analysis For most computer programs, you should format your data the same way you do for an independent-groups t test. The only difference is that if you have, say, four groups, you would code each group as 1, 2, 3, or 4 rather than just 1 or 2. Although full-featured statistics programs such as SAS, SPSS, R, and others can compute Tukey's test, smaller programs (including Analysis Lab) may not. However, these programs are generally able to compute a procedure known as Analysis of Variance (ANOVA). This procedure will be described in detail in a later chapter. Its relevance here is that an ANOVA computes the MSE that is used in the calculation of Tukey's test. For example, the following shows the ANOVA summary table for the "Smiles and Leniency" data. #figure(table( columns: 6, align: left, inset: 6pt, table.header([Source], [df], [SSQ], [MS], [F], [p]), [Condition], [3], [27.5349], [9.1783], [3.4650], [0.0182], [Error], [132], [349.6544], [2.6489], [], [], [Total], [135], [377.1893], [], [], [], )) The column labeled MS stands for "Mean Square" and therefore the value 2.6489 in the "Error" row and the MS column is the "Mean Square Error" or MSE. Recall that this is the same value computed here (2.65) when rounded off. === Tukey's Test Need Not be a Follow-Up to ANOVA Some textbooks introduce the Tukey test only as a follow-up to an analysis of variance. There is no logical or statistical reason why you should not use the Tukey test even if you do not compute an ANOVA (or even know what one is). If you or your instructor do not wish to take our word for this, see the excellent article on this and other issues in statistical analysis by Leland Wilkinson and the APA Board of Scientific Affairs' Task Force on Statistical Inference, published in the #emph[American Psychologist], August 1999, Vol. #emph[54], No. 8, 594–604. === Computations for Unequal Sample Sizes (optional) The calculation of MSE for unequal sample sizes is similar to its calculation in an independent-groups t test. Here are the steps: + Compute a Sum of Squares Error (SSE) using the following formula#math.equation(block: true, alt: "SSE equals ∑ open parenthesis X minus M sub 1 close parenthesis squared plus ∑ open parenthesis X minus M sub 2 close parenthesis squared plus … plus ∑ open parenthesis X minus M sub k close parenthesis squared")[$"SSE" = ∑ ( X − M_(1) )^(2) + ∑ ( X − M_(2) )^(2) + … + ∑ ( X − M_(k) )^(2)$] #linebreak() where M#sub[i] is the mean of the ith group and k is the number of groups. #linebreak() #linebreak() + Compute the degrees of freedom error (dfe) by subtracting the number of groups (k) from the total number of observations (N). Therefore, #linebreak() dfe = N - k. #linebreak() #linebreak() + Compute MSE by dividing SSE by dfe: #linebreak() MSE = SSE/dfe. #linebreak() #linebreak() + For each comparison of means, use the harmonic mean of the n's for the two means (n#sub[h]). #linebreak() All other aspects of the calculations are the same as when you have equal sample sizes.