10.4 Sampling methods

As we mentioned in a previous section, the first thing we should do before conducting a survey is to identify the population that we want to study. Suppose we are hired by a politician to determine the amount of support he has among the electorate should he decide to run for another term. What population should we study? Every person in the district? Not every person is eligible to vote, and regardless of how strongly someone likes or dislikes the candidate, they don't have much to do with him being re-elected if they are not able to vote.

What about eligible voters in the district? That might be better, but if someone is eligible to vote but does not register by the deadline, they won't have any say in the election either. What about registered voters? Many people are registered but choose not to vote. What about "likely voters?"

This is the criteria used in much political polling, but it is sometimes difficult to define a "likely voter." Is it someone who voted in the last election? In the last general election? In the last presidential election? Should we consider someone who just turned 18 a "likely voter?" They weren't eligible to vote in the past, so how do we judge the likelihood that they will vote in the next election?

In November 1998, former professional wrestler Jesse "The Body" Ventura was elected governor of Minnesota. Up until right before the election, most polls showed he had little chance of winning. There were several contributing factors to the polls not reflecting the actual intent of the electorate:

Ventura was running on a third-party ticket and most polling methods are better suited to a two-candidate race.
Many respondents to polls may have been embarrassed to tell pollsters that they were planning to vote for a professional wrestler.
The mere fact that the polls showed Ventura had little chance of winning might have prompted some people to vote for him in protest to send a message to the major-party candidates.

But one of the major contributing factors was that Ventura recruited a substantial amount of support from young people, particularly college students, who had never voted before and who registered specifically to vote in the gubernatorial election. The polls did not deem these young people likely voters (since in most cases young people have a lower rate of voter registration and a turnout rate for elections) and so the polling samples were subject to sampling bias: they omitted a portion of the electorate that was weighted in favor of the winning candidate.

Note

Sampling bias

A sampling method is biased if every member of the population doesn’t have equal likelihood of being in the sample.

import numpy as np

rng = np.random.default_rng(0)     # <-- CHANGE the 0 to re-roll

# The Ventura upset, in miniature. Young voters back the third-party candidate
# heavily, and the "likely voter" screen almost never reaches them.
YOUNG_SHARE       = 0.20   # <-- CHANGE ME: fraction of the electorate that is young
YOUNG_SUPPORT     = 0.80   # <-- their support rate
OLD_SUPPORT       = 0.30   # <-- everyone else's support rate
FRAME_REACH_YOUNG = 0.10   # <-- how often the biased frame reaches a young voter
                           #     set this to 1.00 to un-bias the frame and re-run

N = 100_000
young    = rng.random(N) < YOUNG_SHARE
supports = rng.random(N) < np.where(young, YOUNG_SUPPORT, OLD_SUPPORT)
truth    = supports.mean()

in_frame = rng.random(N) < np.where(young, FRAME_REACH_YOUNG, 1.0)
frame    = np.nonzero(in_frame)[0]          # the list the pollster can actually call

print("TRUE support in the electorate      : %.1f%%" % (truth * 100))
print("Young voters are %.0f%% of the electorate but only %.0f%% of the calling frame."
      % (YOUNG_SHARE * 100, young[frame].mean() * 100))
print()
REPS = 20        # each pollster below runs the same poll 20 times over
print("%8s   %-24s   %-24s" % ("", "fair sample of everyone", "sample from biased frame"))
print("%8s   %9s %6s %7s   %9s %6s %7s"
      % ("n", "avg est", "err", "wobble", "avg est", "err", "wobble"))
print("-" * 70)
for n in (50, 200, 1000, 5000):
    fair = np.array([supports[rng.choice(N, n, replace=False)].mean() for _ in range(REPS)])
    bias = np.array([supports[rng.choice(frame, n, replace=False)].mean() for _ in range(REPS)])
    print("%8d   %8.1f%% %+6.1f %6.1f   %8.1f%% %+6.1f %6.1f"
          % (n, fair.mean() * 100, (fair.mean() - truth) * 100, fair.std() * 100,
                bias.mean() * 100, (bias.mean() - truth) * 100, bias.std() * 100))
print("-" * 70)
print("'wobble' is how much the estimate jumps around between repeat polls.")
print()
print("Both columns' wobble shrinks with n -- that is what more data buys you.")
print("But only the fair sample's ERROR shrinks. The biased frame just clusters ever")
print("more tightly around the WRONG number. No sample size fixes a frame that never")
print("had a chance of including part of the population; set FRAME_REACH_YOUNG = 1.00")
print("and re-run to watch the right-hand column snap onto the truth.")

So even identifying the population can be a difficult job, but once we have identified the population, how do we choose an appropriate sample? Remember, although we would prefer to survey all members of the population, this is usually impractical unless the population is very small, so we choose a sample. There are many ways to sample a population, but there is one goal we need to keep in mind: we would like the sample to be representative of the population.

Returning to our hypothetical job as a political pollster, we would not anticipate very accurate results if we drew all of our samples from among the customers at a Starbucks, nor would we expect that a sample drawn entirely from the membership list of the local Elks club would provide a useful picture of district-wide support for our candidate.

One way to ensure that the sample has a reasonable chance of mirroring the population is to employ randomness. The most basic random method is simple random sampling.

In practice, computers are better suited for this sort of endeavor than millions of slips of paper and extremely large headgear.

It is always possible, however, that even a random sample might end up not being totally representative of the population. If we repeatedly take samples of 1000 people from among the population of likely voters in the state of Washington, some of these samples might tend to have a slightly higher percentage of Democrats (or Republicans) than does the general population; some samples might include more older people and some samples might include more younger people; etc. In most cases, this sampling variability is not significant.

To help account for variability, pollsters might instead use a stratified sample.

Example 2

Suppose in a particular state that previous data indicated that the electorate was comprised of 39% Democrats, 37% Republicans and 24% independents. In a sample of 1000 people, they would then expect to get about 390 Democrats, 370 Republicans and 240 independents. To accomplish this, they could randomly select 390 people from among those voters known to be Democrats, 370 from those known to be Republicans, and 240 from those with no party affiliation.

import numpy as np

rng = np.random.default_rng(0)      # <-- CHANGE the 0 to re-roll

# The book's state: 39% D, 37% R, 24% independent.
# CHANGE the shares (they must sum to 1) or the support rates and re-run.
strata  = ["Democrat", "Republican", "independent"]
share   = np.array([0.39, 0.37, 0.24])
support = np.array([0.75, 0.20, 0.50])     # support for our candidate, by stratum

truth = float(share @ support)
n, REPS = 1000, 4000                       # <-- CHANGE n and watch both columns tighten

# simple random sample: the party mix is itself random, so it wobbles poll to poll
drawn = rng.multinomial(n, share, size=REPS)
srs = rng.binomial(drawn, support).sum(axis=1) / n

# stratified: force the mix to 390 / 370 / 240, then sample inside each stratum
quota = np.round(share * n).astype(int)
strat = rng.binomial(np.tile(quota, (REPS, 1)), support).sum(axis=1) / quota.sum()

print("TRUE support: %.2f%%   (= %s)" % (truth * 100,
      " + ".join("%.2f*%.2f" % (s, p) for s, p in zip(share, support))))
print("Stratified quotas for n=%d: %s"
      % (n, ", ".join("%d %s" % (q, s) for q, s in zip(np.round(share * n).astype(int), strata))))
print()
print("%-22s %10s %10s %10s" % ("method", "mean est", "std dev", "worst miss"))
print("-" * 56)
for label, a in (("simple random", srs), ("stratified", strat)):
    print("%-22s %9.2f%% %9.3f%% %9.2f%%"
          % (label, a.mean() * 100, a.std(ddof=1) * 100, np.abs(a - truth).max() * 100))
print("-" * 56)
print("(%s repeat polls each)" % "{:,}".format(REPS))
print("Both are UNBIASED -- the means land on the truth. Stratifying removes the")
print("wobble in the party mix, so the same n buys a %.0f%% tighter estimate."
      % (100 * (1 - strat.std(ddof=1) / srs.std(ddof=1))))
print("Compare that with the biased-frame cell: there, the mean itself was wrong.")

Stratified sampling can also be used to select a sample with people in desired age groups, a specified mix ratio of males and females, etc. A variation on this technique is called quota sampling.

You may have had the experience of being called by a telephone pollster who started by asking you your age, income, etc. and then thanked you for your time and hung up before asking any "real" questions. Most likely, they already had contacted enough people in your demographic group and were looking for people who were older or younger, richer or poorer, etc. Quota sampling is usually a bit easier than stratified sampling, but also does not ensure the same level of randomness.

Another sampling method is cluster sampling, in which the population is divided into groups, and one or more groups are randomly selected to be in the sample.

Other sampling methods include systematic sampling.

Perhaps the worst types of sampling methods are convenience samples and voluntary response samples.

Usually voluntary response samples are skewed towards people who have a particularly strong opinion about the subject of the survey or who just have way too much time on their hands and enjoy taking surveys.

Adapted from Math in Society by David Lippman, hosted on LibreTexts (math.libretexts.org) and licensed under CC BY-SA 3.0. Changes were made. License: CC-BY-SA-3.0.