Skip to main content
ABC News
Is a Poll Scientific if it Excludes More Than Half the Population?

Last night, I wrote the following on Twitter:

The way YouGov does online polling is no less scientific than the way Rasmussen does telephone polling.

Let me explain what I mean by that. One definition of how “scientific” a poll is is the percentage of the adult population that it can potentially hope to reach. That isn’t a complete definition, mind you — it’s more of a necessary than a sufficient condition — but it isn’t a useless one. By this definition, Rasmussen’s polling isn’t very scientific: because of certain shortcuts that they take, well over half of the American population will be physically unable to take one of their phone calls.

Rasmussen typically conducts its polling on weeknights, calling between 5 PM and 9 PM over the course of a single evening. They do not call phone numbers back, as most other pollsters do, in the event they don’t get an answer the first time. They don’t call cellphones — only landlines. And they speak to the first person they get on the line if they speak to anybody at all; other polling firms use carefully-designed procedures to randomize the selection of respondent within the household (a typical mechanism is something like asking that the adult with the next birthday come to the phone).

Let’s examine the first of these problems in some detail: that Rasmussen only makes one phone call to each household, and that it typically occurs between 5 and 9 PM on a weeknight. What percentage of adults with landlines will physically be able (without unusual effort or impoliteness) to accept a phone call which is made at some random point between 5 and 9 on a weekday evening?

One way to approach this is to look at the American Time Use Survey, which is put out periodically by the Bureau of Labor Statistics. It asks respondents to chart how they spend their time — everything from commuting to bathing — at 10-minute intervals over the course of the day. Some of these activities (like watching TV) are going to be more amenable to taking a pollster’s phone call on one’s landline, and others (like being in the commute home from work) considerably less so.

From the beautiful interactive graphic that my soon-to-be-colleagues at the New York Times put together, I took the average of the number of Americans aged 15 and up who are engaged in a particular activity at 5 PM, 7 PM, and 9 PM respectively, and then made a further adjustment to differentiate between weekends and weekdays. We can pair this time-use data with some commonsensical assumptions about how compatible each activity is with accepting a landline call to get a reasonable estimate of how much of the population Rasmussen can even hope to reach.

The activities were broken down by the Times into roughly 20 categories: for each one, I have attempted to estimate the fraction of those engaged in the activity who are “reachable” by telephone. The definition of “reachable” is unavoidably somewhat ambiguous, but basically refers to people who (i) are within the vicinity of their household’s landline telephone; (ii) are physically able to take the phone call without undue effort; (iii) are not engaged in other activities, like sex, during which one would not customarily answer the phone. The activity categories, going in order from most common to least common, are as follows:

Watching TV or movies (24.0 percent of adult population): You probably won’t have much luck getting someone to take your phone call during the series finale of Lost or the seventh game of the NBA Finals — but the vast majority of people watching the tube are reachable on their landlines by any reasonable definition. The only reason I don’t put this at 100 percent because some TV (and certainly some movie) viewing will occur outside the home, but we’ll assume that … 95 percent are reachable.

Work (13.4 percent of adult population): Although some people work from home, the vast majority will not be available on their home landlines while working … 10 percent are reachable.

Commuting (10.2 percent of adult population): By definition, this activity occurs outside the household … 0 percent are reachable.

Household chores (9.5 percent of adult population): This is a tricky one: about 20 or 25 percent of household tasks occur outdoors, according to the study, and some other activities will be extremely physically strenuous or necessitate that the person quite literally has their hands full. But other chores are more casual (even verging on relaxing) and the person might be quite happy to take a phone call … 70 percent are reachable.

Eating and drinking (7.4 percent of adult population): Obviously, people eating (or drinking) outside their homes won’t be reachable on their landlines. And there are still a lot of people who either turn the phone off or otherwise won’t take calls during dinnertime. Then again, our eating habits are becoming quite casual, and so this rule isn’t as ironclad as it once was. This gets into a little bit of a gray area in terms of how we define the term “reachable”, but let’s say that … 50 percent are reachable.

Sleeping (5.8 percent of adult population): Some people go to bed early or, like yours truly, have otherwise irregular sleeping habits — and I assume that you won’t have much luck in getting someone to take a robopoll if you’ve awoken them from their slumber… 0 percent are reachable.

Other leisure (5.1 percent of adult population): This includes other indoor activities like reading and video games — usually a pretty good time to reach someone … 100 percent are reachable.

Socializing (4.6 percent of adult population): This category is fairly broadly defined and doesn’t necessarily have to involve being outside of one’s household, although a good fraction of it surely will. As with dining, however, this is a time when a lot of people will refuse to take a stranger’s phone call even when they are physically in their homes. … 40 percent are reachable.

Family care (3.9 percent of adult population): These activities will usually, although not always, occur inside the household, but much of it involves being physically in contact with another family member, especially a child, when it might or might not be realistic to take a phone call. And this is another time where people might typically abhor or ignore phone calls. …. 40 percent are reachable.

Personal care (3.3 percent of adult population): A lot of this involves fairly non-strenuous activity like (say) clipping one’s toenails, but three important exceptions are showering, sex, and going to the bathroom, when you either cannot reach the phone or darned well had better not be trying to … 50 percent are reachable.

Sports and exercise (2.4 percent of adult population): The majority of this category, which includes attending sporting events as well as participating in athletic activity, occurs outdoors or entirely outside of the household, and much of what doesn’t will nevertheless involve physical exertion that is incompatible with taking a phone call … 25 percent are reachable.

Education (2.1 percent of adult population): Although most of the time in this category involves attending class, or commuting to and from it, it also includes time spent doing homework, when someone might be willing to take a call during a study break … 30 percent are reachable.

Shopping (1.9 percent of adult population): It’s no longer literally true that this must occur outside the household, but the vast majority of it does … 5 percent are reachable.

Relaxing / thinking (1.6 percent of adult population): i.e., doing nothing … 100 percent are reachable.

E-mail and web surfing (1.1 percent of adult population): If you’re wondering why this number is low, some computer use seems to get classified under “other leisure”. Anyway, this is another very good time to reach someone on the phone … 100 percent are reachable.

Taking phone calls (1.0 percent of adult population): Although most people have call waiting, they won’t usually interrupt a call with a friend, family member or co-worker to take one from a stranger … 20 percent are reachable.

Volunteering (1.0 percent of adult population): With rare exception, this will occur outside the household … 10 percent are reachable.

Nonfamily care, out-of-household (0.7 percent of adult population): The way that the BLS defines this category, which includes things like babysitting, it necessarily occurs away from one’s home … 0 percent are reachable.

Religious activities (0.4 percent of adult population): Mostly attending services or praying; you might occasionally be able to reach someone who is engaged in Bible study … 15 percent are reachable.

Other activities (0.7 percent of adult population): About half of this is out-of-household errands, and the other half is “don’t remember”. The latter might be a euphemism for being drunk/high/stoned, in which case getting someone on the phone is dubious to say the least. But let’s give the “don’t remembers” the benefit of the doubt and say that … 50 percent are reachable.

So, let’s add the numbers up. What percentage of people with landlines will in practice be reachable on them at a randomly-selected time between 5 PM and 9 PM on a weekday evening? The answer, if you follow the highlighted column down to the bottom, is only about — actually, very slightly less than half, according to our estimate.

Basically, a pollster is going to get a lot of people who are watching TV — probably about half his sample. Could that potentially skew things in ways that wouldn’t necessarily be resolved by demographic weighting? Of course it could, particularly since there is a lot of news programming on at this time of day. Otherwise, you’ll get people who are reading or surfing the Internet, who are doing chores, or who, in some cases, are willing to interrupt meal time, family time or social time to take a phone call.

If we were very liberal about how we defined “reachable”, the percentage would be somewhat higher than 50 percent, as activities like eating meals or family care, when not being done away from home, fall into something of a gray area. On the other hand, this definition is generous in a lot of ways. If a travelling salesman is watching TV at his hotel room in Dayton having finished his work for the day, he won’t be available on his home landline, but would be counted as such by our estimate.

Another issue, since Rasmussen doesn’t use intra-household selection and talks to the first person they reach on the phone, is that someone who lives in the household with a spouse or teenager who always takes the household’s phone calls will never be reachable. And by a stricter definition, anyone who does not happen to take the phone call first will not be reachable at that given moment in time. The average American household has 2.59 persons, and even if young children essentially never answer the family’s landline, teenagers certainly do, and so this could reduce reachability by a further factor of about two.

A more unambiguous problem is one we have discussed before around here: not everyone has a landline to begin with. About 23 percent of adults do not have access to a landline, another 4 percent have a landline but only take incoming calls on their cellphone, and 2 percent don’t have a telephone of any kind. In total, this is about 30 percent of the population which can’t be even potentially be reached by a polling firm that doesn’t dial cellphones, and the fraction is increasing every year.

So basically, 70 percent of the population has a landline which they at least sometimes answer, and of that 70 percent, about 50 percent will be “reachable” at a random time on a weekday evening. That means that only about 35 percent of the population can even potentially take one of Rasmussen’s phone calls. If you also account for the lack of intra-household selection in their surveys, the percentage of “reachables” would be more on the order of 20 percent.

Then you get into the problems that are intrinsic to all types of polling, like the fact that there are a lot of people who won’t answer phone calls from unknown numbers at all, or who will hang up once they learn it’s a pollster on the other end of the line. These problems for the most part are unavoidable, although a company with human operators should have lower refusal rates.

But a lot of these problems could be averted if Rasmussen were willing to prioritize quality over quantity. They could certainly do callbacks: if you call someone five times during a 3-day interval, the odds are very high that at least once, they will physically be able to take your call on their landline (whether or not they elect to do so). Rasmussen could include a cellphone sample, as SurveyUSA has started to do for some clients. They could probably introduce some form of intra-household selection; it would be tricky to do with an automated script, but it wouldn’t be impossible.

They don’t elect to do these things, however, because it’s not in their business model. In other words … because they’re cheap. That’s not meant to be an ad hominem: it’s the only way to accurately describe the problem.

Maybe Rasmussen’s polling will happen to produce the right results in spite of all this. Historically, in fact, it’s been about average — a little better than average, actually, on balance. In the long-run, though, I think it’s going to get them into some trouble, particularly as some of these issues — fewer and fewer Americans have landlines each year — are continuing to get worse.

To be clear, most of these things have nothing intrinsically to do with their using an automated script. If you had a pollster that used live human operators, but which did blitz polling during a single four-hour period on a single weeknight, which never did callbacks, which did not call cellphones, and which did not use intra-household selection, you’d have basically all of the same problems. Conversely, some other “robopollsters” like SurveyUSA and PPP avoid at least some of these problems because they (or their clients) are willing to pay the money to do so.

Nevertheless, this is why it’s quite accurate to say what I said in my tweet: that the way that YouGov does online polling is no less scientific than the way Rasmussen does telephone polling. Internet penetration, which is rising, is about to cross paths with landline penetration, which is falling: both are on the order of 70-75 percent of the U.S. population right now. Although the way that YouGov recruits its Internet panel is not completely scientific (although much better than something like Zogby Interactive, which doesn’t even have the pretense of being scientific if you read the fine print), it’s no less scientific than what Rasmussen does, given all the shortcuts they take to churn out polling so cheaply and quickly: somewhere between two-thirds and four-fifths of the population will be physically unable to take one of Rasmussen’s phone calls, depending on how you define the problem.

This is not meant to be an endorsement of YouGov — although I like the guys over there and think they’re smart, and I think competently done online polling (i.e. not Zogby Interactive’s approach) would do more good than harm for the industry. But unless T. Boone Pickens or Bill Gates is willing to bankroll a proper, by-the-book, live-operator survey operation, the future of the industry will mostly be in robopolling and online polling. Therefore, it’s time to pay more attention to which firms do this type of polling relatively well and which do it relatively sloppily; ironically, the fact that some in the traditional polling community are phobic about robopolls (and Internet polls) may lead to less scrutiny of some of the other issues that a firm like Rasmussen has.

Another question I’m sure that I’ll get is to what extent the sheer volume of polling that Rasmussen does is liable to have on our forecasts. The answer is that our forecasting products are very carefully calibrated to avoid allowing any one pollster to unduly influence the outcome: our method builds in several hedges against Rasmussen and sees bad things ahead for Democrats in spite of them, not because of them.

But this discussion shouldn’t be about the results that Rasmussen produces, either past, present or future. It should be about the state of the polling industry as a whole, an industry which is increasingly becoming a race to the bottom, and Rasmussen plays a big part in that.

Nate Silver is the founder and editor in chief of FiveThirtyEight.