Skip to main content
Don’t Take Polling Literally, Part #39916

Public Policy Polling found this interesting result in their national survey earlier this week:

More Politics

That’s right: 21 percent of respondents said the spill in the Gulf makes them more likely to support further offshore drilling efforts.

When I tweeted about this the other day, people assumed that I was making fun of the people who selected this response, or perhaps the pollster. But I’m not really doing either.

First, insofar as the pollster goes, this is actually the correct, Polling 101 way to frame the question. You want your questions to be neutral and balanced. You don’t want to lead the respondent toward any particular response or make assumptions about which responses they might consider “reasonable”. For instance, rather than ask “Do you think Barack Obama was born in Kenya, or not?”, you’re better off asking “Do you think Barack Obama was born in Kenya, in the United States, in Indonesia, or somewhere else?”. That’s a balanced question. Importance questions, e.g. “How important is the inclusion of a public option in the health care bill: Very important, somewhat important, or not at all important?”, are hard to balance and should usually be avoided; they almost never tell you anything useful. So the pollster has committed no real fault here.

At the same time, if you do give people an “illogical” response, sometimes they’re going to pick it! This question was immediately preceded by one about support for offshore drilling overall. It may be that some of the respondents had an instinct to dig in their heels and wanted to appear consistent, using the question to reaffirm the support for offshore drilling that they’d already expressed. That is, saying that the spill makes you more likely to support offshore drilling may not literally be true but instead may mean something more along the lines of “you’re damned right I support offshore drilling!”.

It may also be that some people were confused by the question, or were not paying much attention to it. A lot of people might have taken PPP’s call while ironing their clothes, or watching TV, or getting their kids ready for soccer practice. They might have been zoning out, and heard blah-blah-blah-blah oil spill blah-blah-blah-blah more, and thought it was the choice used to express that they had become more concerned — not more supportive — of offshore drilling. There is arguably some evidence of this in that the percentage of people selecting this response feel a bit off and do not always track underlying opinions about offshore drilling. For instance, 17 percent of liberals picked the “more likely” response, even though only 33 percent of them supported offshore drilling to begin with, and 29 percent of African-Americans did, even though just 44 percent of them support offshore drilling.

These issue may be exacerbated by the fact that PPP uses automated polling technology; a human interviewer is more likely to command the full attention of the subject.

By the way, there’s an interesting discussion to be had about whether the spill in the Gulf should make us more concerned about offshore drilling, or whether it was an essentially random event that should already have been priced into our assessments, in the same way that the risk of earthquakes is something you should already be thinking about when you’re trying to decide whether to buy a home in San Francisco and so you shouldn’t flee the city after one occurs. My personal take is that the fact that a spill occurred does not change the equation much, but the fact that it’s been so hard to stop and to contain, and that the public may wind up bearing a lot of a private business’s risks, possibly does.

Nate Silver is the founder and editor in chief of FiveThirtyEight.

Comments Add Comment