Skip to main content
ABC News
Study: Excluding Cellphones Introduces Statistically Significant Bias in Polls

This is a graph, per the CDC, of the percentage of “cellphone-only” adults in the United States. The fraction, as of the second half of 2009, was 23 percent of all U.S. adults, or 25 percent of all U.S. households. These adults — about a quarter of the population — are simply ignored by many pollsters.

This is not a new problem — in fact, it’s one we’ve written about on several occasions. But it’s continuing to get worse. The percentage of people who have replaced their landlines with cellphones has climbed at a remarkably steady rate. (There may have been an especially large leap from the second half of 2008 to the first half of 2009, when the recession was at its worst and many people were looking for ways to trim household costs.)

Bear in mind that these figures are already somewhat out of date. The fraction could be in the high 20s by the time we get to November. And if the current trends hold, it could be in the mid or high 30s by the time we get to 2012. Nor does the figure include so-called “cellphone-mostly” households, which is when the house has a landline, but rarely or never uses it to receive incoming calls; another 15 percent of the population falls into this category.

Cellphone-only households are different from their landline-using counterparts. They tend to be younger, poorer, more urban, less white, and more Internet-savvy. All of these characteristics are correlated with political viewpoints and voting behavior.

The pollsters’ usual defense mechanism against this is to weight their polls by demogrpahics — something which they need to do anyway, since polls are subject to many forms of non-response bias (for instance, it’s harder to get men on the phone then women). But this is potentilly an inadequate response for several reasons. First, some characteristics that correlate with both cellphone usage and political preferences may not correspond to those that are most commonly used to weight polls. It is somewhat rare, for instance, for pollsters to weight their polls by characteristics like urban/rural location or marital status, which are predictive of both cellphone usage and political beliefs. Being cellphone-dependent also appears to be significantly correlated with media consumption habits (in particular, getting more of one’s news from the Internet and less from television), which also seems to be increasingly important in determining one’s political views. And there are some characteristics that may be even more subtle. For instance, there are some hints in the CDC data (such as the higher prevalance of binge drinking) that cellphone-only adults are less “domestic” and more “bohemian”. I suspect that, in young adults, this is correlated with more liberal political views.

Secondly, even where weighting occurs, one may encounter problems when upweighting from very small subsamples. It is now very difficult, for instance, to get young people on the phone when using a landline-only sample. About half of all adults from age 25-29, for instance, are cellphone-only, and two-thirds are either cellphone-only or cellphone-mostly. (The numbers are actually slightly better for adults aged 18-24, who are more likely to be living in a college dormatory, or still to be living at home, where a landline will usually be available.) Couple this with the fact that young people have grown up in a call-screening culture, and their response rates are often completely inadequate. Say that you’re supposed to have 100 people aged 18-29 in a poll of 500 adults, but in fact you only get 30 because of problems with call-screening and cellphone usage. The margin of error on a sample of that size is 18 percent. And yet, you may essentially let each of these young people speak on behalf of two or three of their peers, to compensate for the ones you haven’t gotten in contact with.

A new study from Pew, in fact, has found that these weighting schemes may have become inadequate. In their experiment, a weighted landline-only sample produced a generic ballot result of Republicans 47, Democrats 41, whereas a weighted landline-plus-cellphone sample had the generic ballot tied 44-44. That six-point net difference is statistically significant, and needless to say, could have huge implications for where the parties finish in November.

With that said, I certainly wouldn’t go out and append 6 points to the Democrats’ generic ballot number. For one thing, some pollsters do include cellphones in their sample. For another, the results from Pew reflect just one study/experiment, one which itself is subject to sample bias. Also, Pew’s study finds that cellphone-only adults are less likely to vote, so the differential is probably less in the case of likely voters.

It also may be that this is case of two wrongs making a right. For instance, many pollsters aren’t yet using a likely voter model, which tends to work to the benefit of Democrats since their voters are less likely to turn out, especially in a cycle like this one in which there appears to be an unethusiasm gap. There’s also some evidence that Democrats underform the generic Congressional ballot on election day. If the failure to include cellphones biases the numbers in Republicans’ direction, it may compensate for these other deficiencies. If so, however, it does so in a tenuous way, and there may be other cycles where the biases run in the same direction.

What can the pollsters do to make up for this?

The most obvious solution, of course, is to call cellphones, which more and more of the major-media pollsters (like Gallup, Pew, ABC/Post, CBS/NYT, NBC/WSJ and AP-Gfk) are doing, as are some academic polling centers like Quinnipiac. But this is expensive: calling a cellphone is about twice as expensive as calling a landline, according to Pew. And the cost differential may be greater for pollsters using robodialers, since cellphone numbers are subject to additional FCC restrictions on automated calling.

I have mixed sympathies for the pollsters on this one. On the one hand, traditional media budgets are still being cut, and running a high-quality, traditional polling operation is expensive. On the other, there are a few pollsters who seem to emphasize quantity and speed over methodoligical soundness, and they are unambiguously a part of the Problem.

Another approach, in the absence of calling cellphones, is to increase the sample size that one uses. Although there’s not that much difference between calling an unbiased sample of 500 respondents or 1,000 (the associated margins of error are 4.4 points and 3.1 points, respectively), these differences are magnified if one relies upon upweighting results from smaller subsamples to correct for response bias, such as those for young voters or Hispanics.

Finally, pollsters might want to consider weighting based on “non-traditional” criteria such as urban/rural status, technology usage, or perhaps even media consumption habits.

Polling, like the media environment itself, is becoming increasingly fast, cheap — and arguably, out of control. That many pollsters willingly ignore the full quarter of the population who do not have landlines is but one symptom of the problem.

Nate Silver founded and was the editor in chief of FiveThirtyEight.