In its latest survey, YouGov, an online market-research firm, asked about 1,000 U.S. adults, “Would you personally waterboard a suspected terrorist?” More than a quarter of them — 27 percent — said yes. The results surprised me, but they also made me suspicious.
Self-reporting is problematic by itself. People often inaccurately report their own data, either through inability (forgetting) or unwillingness (feelings such as shame and pride can induce what’s known as “social desirability bias”). But hypothetical questions are even more problematic.
Hypotheticals can work in some cases. Companies spend billions of dollars on focus groups, some of which are designed to figure out whether someone would, hypothetically, buy products or services under different scenarios. But even basic questions that ask a respondent to predict her future behavior — “Do you plan to vote?” — have proved unreliable.
And the less basic and more outlandish a hypothetical question is, the less reliable it can be. Hardly anyone has any experience with waterboarding — doing it, seeing it or even being near it. It’s a safe bet that virtually everyone who answered the YouGov survey based their responses on an imaginary decision-making context — with imaginary suspected terrorists and imaginary feelings toward them.
Michael Traugott, a professor of communication studies and political science at the University of Michigan (and former president of the American Association for Public Opinion Research), told us that the extremity of the hypothetical behavior is likely to affect the accuracy of reporting:
“If you ask a person, ‘When you purchase your next car, will you purchase a blue or red car?'” you can generally be confident in the answer, Traugott said. “People know what color they like.”
But waterboarding “is not a realistic possibility for the average citizen. That lack of realism increases the possibility that respondents would say they would waterboard a suspected terrorist,” Traugott said. “They’re not actually facing the real circumstances.”
Other survey questions have similarly been based on other improbable scenarios. Take, for example, a 2011 study published in the American Journal of Public Health. Participants were asked to choose between being obese or having one of 12 socially stigmatized conditions. Almost 15 percent of women between 18 and 45 years old said they would rather be blind than obese. About a quarter said they would prefer severe depression to obesity.
These questions can be interesting, even revealing, but it’s harder to say whether they elicit accurate answers. The next time you read “would” in a survey question, treat the numbers that follow with caution.