10.5 How to mess things up before you start

There are number of ways that a study can be ruined before you even start collecting data. The first we have already explored – sampling or selection bias, which is when the sample is not representative of the population. One example of this is voluntary response bias, which is bias introduced by only collecting data from those who volunteer to participate. This is not the only potential source of bias.

Sources of response bias may be innocent, such as bad memory, or as intentional as pressuring by the pollster. Consider, for example, how many voting initiative petitions people sign without even reading them.

Loaded questions can occur intentionally by pollsters with an agenda, or accidentally through poor question wording. Also a concern is question order, where the order of questions changes the results. A psychology researcher provides an example[2]:

“My favorite finding is this: we did a study where we asked students, 'How satisfied are you with your life? How often do you have a date?' The two answers were not statistically related - you would conclude that there is no relationship between dating frequency and life satisfaction. But when we reversed the order and asked, 'How often do you have a date? How satisfied are you with your life?' the statistical relationship was a strong one. You would now conclude that there is nothing as important in a student's life as dating frequency.”

Example 6

A telephone poll asks the question “Do you often have time to relax and read a book?”, and 50% of the people called refused to answer the survey. It is unlikely that the results will be representative of the entire population. This is an example of non-response bias, introduced by people refusing to participate in a study or dropping out of an experiment. When people refuse to participate, we can no longer be so certain that our sample is representative of the population.

import numpy as np

rng = np.random.default_rng(0)     # <-- CHANGE the 0 to re-roll

# "Do you often have time to relax and read a book?"  Half the people hang up.
# The catch: the people with no time to relax are also the people with no time
# for a phone survey. Non-response is only harmless when it is unrelated to the answer.
N = 50_000
has_time = rng.random(N) < 0.40           # <-- TRUE rate: 40% do have time to relax
truth = has_time.mean()

print("TRUE answer in the population: %.1f%% say yes" % (truth * 100))
print()
TARGET = 0.50      # <-- every row below is pinned to this overall response rate
print("%-36s %9s %11s %9s" % ("who answers the phone", "resp rate", "poll says", "error"))
print("-" * 70)

for ratio in (1.0, 1.5, 2.5, 5.0):     # <-- CHANGE these: how much likelier a
    # solve for the two rates that keep the OVERALL response rate at TARGET
    p_no  = TARGET / (truth * ratio + (1 - truth))
    p_yes = min(1.0, ratio * p_no)
    responds = rng.random(N) < np.where(has_time, p_yes, p_no)
    measured = has_time[responds].mean()
    label = ("everyone equally likely to answer" if ratio == 1
             else "relaxed folk %.1fx likelier to answer" % ratio)
    print("%-36s %8.0f%% %10.1f%% %+9.1f" % (label, responds.mean() * 100,
                                             measured * 100, (measured - truth) * 100))
print("-" * 70)
print("Every row has the SAME overall response rate (%.0f%%) and about the same" % (TARGET * 100))
print("number of completed interviews. Only the first one is trustworthy.")
print()
print("A 50% response rate is not the problem by itself -- WHO is missing is.")
print("That is why 'we surveyed 50,000 people' is not an answer to 'who refused?'")

Your Turn

Try it Now 5

In each situation, identify a potential source of bias

A survey asks how many sexual partners a person has had in the last year
A radio station asks readers to phone in their choice in a daily poll.
A substitute teacher wants to know how students in the class did on their last test. The teacher asks the 10 students sitting in the front row to state their latest test score.
High school students are asked if they have consumed alcohol in the last two weeks.
The Beef Council releases a study stating that consuming red meat poses little cardiovascular risk.
A poll asks “Do you support a new transportation tax, or would you prefer to see our public transportation system fall apart?”

Answer

Response bias – historically, men are likely to over-report, and women are likely to under-report to this question.
Voluntary response bias – the sample is self-selected
Sampling bias – the sample may not be representative of the whole class
Lack of anonymity
Self-interest study
Loaded question

[1] Reuters. news.yahoo.com/s/nm/20090423/...k_gum_learning. Retrieved 4/27/09

[2] Swartz, Norbert. www.umich.edu/~newsinfo/MT/01...01/mt6f01.html. Retrieved 3/31/2009

Adapted from Math in Society by David Lippman, hosted on LibreTexts (math.libretexts.org) and licensed under CC BY-SA 3.0. Changes were made. License: CC-BY-SA-3.0.