Skip to main content
ABC News
Automated Poll Produces Starkly Different Results on Gay Marriage Question

Not so fast, ladies and ladies (and gentlemen and gentlemen). On the heels of a CNN poll earlier this week which was the first ever to show majority support for gay marriage, Public Policy Polling has come out with a survey showing a clear majority of their respondents still opposed to it: 57 percent of registered voters who responded to their survey think gay marriage should be illegal, and 33 say it should be legal. This contrasts sharply with CNN’s results, which showed either a 52-46 majority in support of gay marriage or a narrow 48-50 plurality opposed to it, depending on the question wording. In fact, the 33 percent PPP shows in support of gay marriage is the lowest of any poll since 2006.

There are several factors that might account for the differences:

Different sample frames. PPP’s poll is of registered voters, whereas CNN’s was conducted among all adults. Since adults who are not registered to vote tend to be younger, and younger people tend to be more supportive of gay marriage, that might account for a couple points’ worth of difference. However, this might be counteracted to some extent by the fact that minorities are also more likely to be unregistered, and African-Americans tend to show demonstrably less support for gay marriage (it is not clear that this is true of Hispanics or Asians.)

Different question wording. PPP’s poll asked whether gay marriage should be legal or illegal; CNN’s asked whether there is currently, and whether there should be, a Constitutional right to it. These are somewhat different questions both in theory and especially in practice given the strong feelings that Americans have about the Constitution.

Sample variance. In other words, random noise. PPP’s poll consisted of 606 voters, a relatively small sample; CNN’s consisted of 1,000 adults, which is not much larger, and their sample was split into halves by the two forms of the question that they posed. Before the CNN and PPP polls came out, the trendline pointed to around ~44-45 percent support for gay marriage: it is possible, and perhaps somewhat likely, that both polls are statistical outliers on either side of this trend.

There is, however, somewhat more evidence that the PPP poll has suffered from sample anomalies. In particular, the age distribution they show is rather flat: just 44 percent of 18-29 year-old respondents said they supported gay marriage, versus 31 percent of registered voters aged 66 and up. By contrast, a Pew poll in August, 2009, which had a much larger (~2,000 person) sample and for which comprehensive cross-tabs are available , had 58 percent of 18-29 year-olds in support of gay marriage, but just 20 percent of those 66 and older. Likewise, the Proposition 8 exit poll in California in 2008, which also had a larger sample, had 61 percent of 18-to-29 year olds opposed to Prop 8 (that is, supporting gay marriage) versus 39 percent of those 65 and up.

Moreover, the PPP poll has only 11 percent of its respondents between ages 18-29, whereas 19 percent of actual voters in 2008 were. And they show essentially no racial split in support for gay marriage, which contradicts virtually all other research on the topic.

Part of this probably reflects the low response rates associated with automated surveys, as well as the fact that they don’t call cellphones (although, to my knowledge, CNN does not currently poll cellphone voters either). Young people are hard to get on the phone, and the ones that you do get on the phone may not be especially representative of their cohort. It is hard to believe that a majority of Americans under the age of 29 think gay marriage should be illegal.

Automated versus live-operator. PPP’s was, to my awareness, the first automated survey (“robopoll”) that took a fair shot at asking the gay marriage question, notwithstanding a Rasmussen poll in 2006 that asked somewhat leadingly about the “definition of marriage”. CNN’s poll, by contrast, used live operators. There has been some speculation that people are apt to answer more honestly on delicate issues like gay rights when probed by a robopoll rather than a live-operator survey. Since it has become somewhat “politically incorrect” to oppose gay rights, it’s possible that the automated surveys are relatively more immune from social desirability bias.

While there may be some truth to this, I don’t think it entirely explains PPP’s results. The reason is that their poll also showed fairly low levels of support for marijuana legalization: 34 percent in favor and 52 percent opposed. While that’s not an enormous outlier, the average of the four live-operator polls on marijuana legalization since the start of 2010 have shown an average of 41 percent support for its legalization.

As I explained at length here, you’d expect social desirability bias on the marijuana question to run the other way, i.e., people might be more willing to express support for legalizing marijuana to an automated script rather than to a real person on the other end of the line who might be a mother, an impressionable teenager, etc. And indeed, there is some evidence that marijuana rights poll better on automated surveys. But this one was an exception, which leads me to wonder whether it simply drew a non-representative sample.


I mentioned yesterday that the graph we produced, which appeared to show accelerating support for gay marriage, was quite sensitive to new polling data on the endpoints. If we include the PPP poll in the graph, and re-run the LOESS regression, we no longer show an accelerating trend toward support for gay marriage but instead, a steady-as-she-goes one, with support currently on the order of 43-44 percent and opposition at about 52 percent. The lines are converging at a rate of about 1.5 points per year which means that they would cross at some point in 2013, 2014, or thereabouts.

In a lot of ways, this is probably the more neutral hypothesis on support for gay marriage, particularly given that, as PPP’s Tom Jensen notes, gay marriage initiatives recently failed, albeit narrowly, in blue states like Maine and California. (Although bear in mind that the California and Maine results reflect only people who voted, and not all adults in those states.)

At the same time, the graph I posted yesterday is arguably a more apples-to-apples comparison, since PPP’s is the only automated survey on this subject. If we look at trendlines in the individual surveys, by contrast, CNN and Gallup have both shown increases in support for gay marriage of about 4 points over the past year. While I have nothing against automated surveys in general, and there is something to be said for their capacity to avoid social desirability bias, they are far less tested when it comes to measuring support for policy questions, rather than producing “horse race” numbers. Meanwhile, some of the cross-tabs in the PPP poll are dubious.

Here’s what I think it’s safe to say: it is dangerous, and probably even a little irresponsible, to say “Americans think so-and-so” based on the results of one individual survey — especially when it’s your survey. Jensen, for instance, headlined his article “Americans still opposed to gay marriage”. That’s a little presumptuous, particularly when you are not calling unregistered voters, and are not calling anyone who uses a cellphone rather than a landline, which will disqualify around 45 percent of the American population — and when most of the remaining 55 percent will hang up once they recognize its a pollster calling. Statements like these are even more dubious when they come from a pollster like Rasmussen, which takes far more shortcuts than PPP does and probably excludes 75 or 80 percent of Americans from even having the opportunity to answer one of their surveys.

By all means, there should be more polling on gay marriage, including from automated survey firms. But particularly when your survey produces results that are “different” from the consensus — this is arguably true of the CNN poll, and certainly true of the PPP poll — you should perhaps go back and do some additional diligence, whether that means going into the field with a larger sample size, asking the questions in a different way, or paying for a cellphone sample, rather than proclaiming what may be a bug to be a feature.

UPDATE: A FOX News poll, also out today, shows increasing levels of support for gay marriage since 2009. Giving their respondents a three-way choice of marriage, civil unions, and no legal recognition, they find numbers of 37/29/28, with marriage getting the plurality. A year ago, their numbers were at 33/33/29.

Nate Silver founded and was the editor in chief of FiveThirtyEight.