10.6 Experiments

So far, we have primarily discussed observational studies – studies in which conclusions would be drawn from observations of a sample or the population. In some cases these observations might be unsolicited, such as studying the percentage of cars that turn right at a red light even when there is a “no turn on red” sign. In other cases the observations are solicited, like in a survey or a poll.

In contrast, it is common to use experiments when exploring how subjects react to an outside influence. In an experiment, some kind of treatment is applied to the subjects and the results are measured and recorded.

Here are some examples of experiments:

When conducting experiments, it is essential to isolate the treatment being tested.

The difficulty with this scenario is that it is not clear whether the curriculum is responsible for the improvement, or whether the improvement is due to a math specialist teaching the class. This is called confounding – when it is not clear which factor or factors caused the observed effect. Confounding is the downfall of many experiments, though sometimes it is hidden.

Note

Confounding

Confounding occurs when there are two potential variables that could have caused the outcome and it is not possible to determine which actually caused the result.

import numpy as np

rng = np.random.default_rng(0)      # <-- CHANGE the 0 to re-roll

# The book's middle school: a new curriculum AND a hired math specialist,
# both introduced in the same classroom. Scores go up. What caused it?
CURRICULUM_EFFECT = 0.0    # <-- CHANGE ME. Start at 0: the curriculum does NOTHING.
SPECIALIST_EFFECT = 8.0    # <-- the specialist really is a better teacher
BASE, NOISE, N    = 62.0, 6.0, 2000

def run(assign_specialist):
    """assign_specialist(new_curriculum) -> whether that class also got the specialist"""
    new_cur   = rng.random(N) < 0.5
    specialist = assign_specialist(new_cur)
    score = (BASE + CURRICULUM_EFFECT * new_cur + SPECIALIST_EFFECT * specialist
             + rng.normal(0, NOISE, N))
    return score[new_cur].mean() - score[~new_cur].mean()

# Design 1 (the school's): the specialist teaches EXACTLY the new-curriculum class.
confounded = run(lambda new_cur: new_cur)
# Design 2: the specialist is assigned at random, independent of the curriculum.
randomized = run(lambda new_cur: rng.random(N) < 0.5)

print("Truth we built into the simulation:")
print("   new curriculum is worth %+.1f points" % CURRICULUM_EFFECT)
print("   the math specialist is worth %+.1f points" % SPECIALIST_EFFECT)
print()
print("%-46s %12s" % ("design", "measured gap"))
print("-" * 60)
print("%-46s %+11.1f" % ("specialist teaches the new-curriculum class", confounded))
print("%-46s %+11.1f" % ("specialist assigned at random (controlled)", randomized))
print("-" * 60)
print()
print("The confounded design credits the curriculum with about %+.1f points" % confounded)
print("even though we set its true effect to %+.1f. It is measuring the specialist." % CURRICULUM_EFFECT)
print("Randomly assigning the second variable breaks the tie between them and the")
print("measured gap collapses to the curriculum's real effect.")
print()
print("Now set CURRICULUM_EFFECT = 5.0 and re-run: the confounded design reports")
print("about 13 points, and you still cannot tell which 5 came from where.")

There are a number of measures that can be introduced to help reduce the likelihood of confounding. The primary measure is to use a control group.

Ideally, the groups are otherwise as similar as possible, isolating the treatment as the only potential source of difference between the groups. For this reason, the method of dividing groups is important. Some researchers attempt to ensure that the groups have similar characteristics (same number of females, same number of people over 50, etc.), but it is nearly impossible to control for every characteristic. Because of this, random assignment is very commonly used.

Sometimes not giving the control group anything does not completely control for confounding variables. For example, suppose a medicine study is testing a new headache pill by giving the treatment group the pill and the control group nothing. If the treatment group showed improvement, we would not know whether it was due to the medicine in the pill, or a response to have taken any pill. This is called a placebo effect.

To control for the placebo effect, a placebo, or dummy treatment, is often given to the control group. This way, both groups are truly identical except for the specific treatment given.

In some cases, it is more appropriate to compare to a conventional treatment than a placebo. For example, in a cancer research study, it would not be ethical to deny any treatment to the control group or to give a placebo treatment. In this case, the currently acceptable medicine would be given to the control group, sometimes called a comparison group in this case. In our SAT test example, the non-treatment group would most likely be encouraged to study on their own, rather than be asked to not study at all, to provide a meaningful comparison.

When using a placebo, it would defeat the purpose if the participant knew they were receiving the placebo.

It should be noted that not every experiment needs a control group.

[1] Levine JD, Gordon NC, Smith R, Fields HL. (1981) Analgesic responses to morphine and placebo in individuals with postoperative pain. Pain. 10:379-89.

Adapted from Math in Society by David Lippman, hosted on LibreTexts (math.libretexts.org) and licensed under CC BY-SA 3.0. Changes were made. License: CC-BY-SA-3.0.