Mark Blumenthal has the details on a budding controversy between Stan Greenberg, the co-founder of the Democratic polling organization Democracy Corps, and Ed Gillespie and Whit Ayres, the co-founders of the new Republican polling form Resurgent Republic. As Blumenthal notes, there is a bit of irony here, as Resurgent Republic was founded with the explicit mission of mirroring Democracy Corps’ role at the nexus of testing public opinion and, as they put it, “shaping the debate over the proper role of government”.
Greenberg’s main beef is that Resurgent Republic’s sample in their first national survey (.pdf) includes an implausibly high number of Republicans. He writes, in a letter addressed to Gillespie:
…I am perplexed that your first poll would be so outside the mainstream on partisanship. Your poll gives the Democrats just a 2-point party identification advantage in the country, but other public polls in this period fell between +7 and +16 points – giving the Democrats an average advantage of 11 points. Virtually all your issue debates in the survey would have tilted quite differently had the poll been 9 points more Democratic.
[…] If the Resurgent Republic poll is to be an outlier on partisanship, then I urge you to explain what about your methodology produces it – or simply to note the difference in your public release.
The 2-point partisan advantage that Resurgent Republic gives Democrats is the number you get if you include independent “leaners” in the tally. Without the leaners, as Ayres noted to Talking Points Memo, the Democratic advantage is 4 points. This is a matter of taste, I suppose, but I am inclined to give Ayres the benefit of the doubt here. When I was compiling data on the partisan ID gap last week, I treated “independent, lean Democrat” voters and “independent, lean Republican” voters as nevertheless being independents, which is the same practice that Pollster.com has adopted in its partisan ID tracking.
So let’s evaluate a 4-point partisan ID advantage for the Democrats: is it proper to term this an “outlier”, as Greenberg does?
The chart below provides the Democrats’ partisan ID advantage from among the 19 national pollsters who have tested this question at least once within the last 60 or so days. The values run from an 18-point advantage for the Democrats on the high end, as indicated on the most recent CBS/NYT survey, to the 4-point advantage that Resurgent Republic finds:
Resurgent Republic, as we mentioned, finds the smallest Democratic advantage from among the 19 pollsters who tested the question. But, this does not necessarily make them an outlier. One thing to notice is that the bars in the chart above are color-coded. This time of year, when we’re not in the midst of an election cycle, pollsters are pretty evenly divided between those who are testing likely voters, those who are testing registered voters, and those who are testing all adults. (Personally, if I were running a polling firm, I would lean toward all adults when this far removed from an election cycle, but that is a subject for another day).
The different sample frames are associated with different degrees of advantage for the Democrats:
Average Democratic Partisan ID Advantage
All Adults (10 surveys) 11.8 points
Registered Voters --
without Resurgent Republic (5 surveys) 9.8 points
with Resurgent Republic (6 surveys) 8.8 points
Likely Voters (3 surveys) 7.3 points
-----------------------------------------------------
All Surveys --
without Resurgent Republic (18 surveys) 10.5 points
with Resurgent Republic (19 surveys) 10.2 points
Democrats have the largest partisan ID advantage — nearly 12 points — in the surveys that test all adults. This is followed by the registered voter surveys, where Democrats have a 9.8 point advantage without Resurgent Republic’s results and 8.8 points with them, and then by the likely voter surveys, where they have a 7.3-point advantage (closely matching their 7-point edge on the 2008 national exit poll). This comports with the conventional wisdom, which holds that Democrats do proportionately better with “unlikely” or unregistered voters, who tend to be younger and less white (and therefore more Democratic, given recent trends) than their likely voter counterparts.
The point is that, if we’re evaluating Resurgent Republic’s results, we should compare them against other pollsters who applied the same sample frame, which is registered voters. The five other pollsters that looked at registered voters (FOX, PPP, Diageo/Hotline, Cook/RT and National Journal) gave the Democrats an average advantage of 9.8 points. This compares with the 4-point advantage — actually, 4.5 points — that Resurgent Republic gave them.
If we assume that the 9.8-point value is the “true” and correct result, then a 4.5 point advantage would not be all that out of line. Resurgent Republic’s reported margin of error, given its 1,000-person sample size, is 3.1 points. However, this reflects the margin of error for one category only, such as the number of Democrats OR the number of Republicans, and not the difference between the two. The matter gets a bit more complicated still because Resurgent Republic in fact includes a third category — independents. But if we assume that the “true” distribution of registered voters in the population is, say, 37.4 percent Democrat, 27.6 percent Republican and 35.0 percent independent (giving the Dems a 9.8-point advantage), the odds of the Democrats winding up with no more than a 4.5-point advantage in a 1,000-person sample is about 2 percent due to chance alone — somewhat unlikely, but far from impossible. Moreover, since we do not have any a priori way of knowing the “true” number of registered Democrats and registered Republicans (many states either do not report registration figures, or do not require voters to register by party at all), it becomes harder still to characterize Resurgent Republic’s results as an outlier.
What we can say, however, is that if Resurgent Republic consistently shows a smaller partisan ID gap than other pollsters, it will become progressively harder — and will eventually become entirely impossible — to attribute these results to chance alone. Instead, one of two things will become true: either Resurgent Republic is (intentionally or unintentionally) doing something wrong, or Resurgent Republic is right and everyone else is wrong (in which case, they had better be prepared to defend their methodology). So far, however, we can not say that Resurgent Republic has a consistent pattern of producing a sample frame favorable to Republicans because they do not have a pattern at all — this was their first survey.
I suppose we could get all Bayesian and debate the probability that, given that Resurgent Republic is a proudly Republican pollster, they somehow put their finger on the scale, instead of letting the results fall as they might (and just so happening to come up with a Republican-leaning sample). But I don’t particularly see what that exercise would accomplish. Resurgent Republic will presumably release many more surveys over the coming months and years: if they consistently show a Republican-leaning house effect then, believe you me, we’ll be screaming from the rooftops about it. But so far, again, they have done nothing like this, because they only have one survey to their credit.
Another interesting portion of Greenberg’s critique is when he discusses the marketing implications of Democracy Corps’ philosophy on sampling:
One thing Democracy Corps has tried to do is be very “conservative” – watching very closely to make sure all our choices in survey design are well grounded or tilted against the Democrats, including the choice of “likely voters” that normally favors the Republicans. You have probably noticed that our job approval ratings for George Bush were almost always higher than the average of polls, just as our job approval ratings for Barack Obama are now somewhat lower.
Greenberg is right that Democracy Corps’ horse race and approval rating polling has not shown a particular Democratic lean — in fact, their horse race numbers last year had a slight (although not statistically significant) Republican lean. Should Ayres and Resurgent Republic follow his lead?
From a marketing perspective, they might do well to follow Greenberg’s advice. Democracy Corps has built up a strong (and, in my opinion, well-deserved) reputation for accuracy in spite of their partisan affiliation.
But from a scientific perspective, I’d hope that Resurgent Republic simply tried to be as fair as possible and let their results speak for themselves. If Resurgent Republic did design a fair and unbiased survey, and literally just so happened to come up with results that included an unusually high number of Republicans (or for that matter, an unusually high number of Democrats), as will happen from time to time, I’d certainly hope that they reported those results as is. The very last thing I’d want to see a pollster do is to not report data because it was “politically incorrect” or otherwise bad for business for them to do so. SurveyUSA, which is a rather good pollster, at one point last year came out with a poll that gave John McCain a 20-point lead in North Carolina (a state which Barack Obama eventually won). SurveyUSA got a lot of crap for this, and it was clear at the time that the result was reasonably likely to turn out to be an outlier, but it’s to their credit that they reported the result for what it was. Once a pollster starts to self-censor, we wind up with little more than their prefabricated opinions about what the world looks like.
I’ve largely defended Resurgent Republic here, but two other, more skeptical comments. Firstly, the most unorthodox result in their poll was not the Democrat-Republican gap, but the conservative-liberal gap. 47 percent of their respondents described themselves as “very conservative” or “somewhat conservative”, versus just 23 who described themselves as “somewhat liberal” or “very liberal”. This is a much larger advantage than conservatives had on the national exit poll, when 34 percent of voters described themselves as conservative versus 22 percent liberal (although, the results aren’t directly comparable because Resurgent Republic uses the “very” and “somewhat” qualifications, whereas the national exit survey did not).
Secondly — and this is something that Greenberg caught too — I think Resurgent Republic impaired their credibility by inserting the loaded phrase “spreading the wealth” into a question about the economy; using one of Joe The Plumber’s favorite catch phrases is not a good way to perpetuate the notion that you’re acting in good faith.
Still, these are venal rather than cardinal sins, particularly for a virgin survey from a virgin pollster. I don’t know Ayres or his partner Ed Gillespie, but they seem like sincere enough fellows, and their survey was excellent on disclosure and transparency issues. Presumably they are also smart enough to know that spin tends to get called out for what it is these days.
This website too, after all, is founded on the premise that one can have one’s cake and eat it too — that one can do good, fair, scientific work, while nevertheless being forthright about one’s position along the political spectrum. I hope that I’ve earned and maintained your trust, and I wish Resurgent Republic the best of luck in earning it too.