FiveThirtyEight

Politics

Although I expect to proceed fairly carefully with respect to Research 2000, which Daily Kos will be suing for alleged fraud, I have suggested here and to at least one reporter that I had my own suspicions about Research 2000 which paralleled some of the findings in the study by Mark Grebner, Michael Weissman, and Jonathan Weissman. I want to be a bit more explicit about what I mean by that.

This is a copy of two e-mails that I sent to Mark Blumenthal of Pollster.com in the wee hours of the morning on February 4th. Like the examples in the Grebner study, they point toward cases in which Research 2000’s data appeared to be other-than-random (although, as I declaim in the e-mails, not necessarily triggered by fraud).

from Nate Silver [xxx@xxx]to Mark Blumenthal [xxx@xxx]Mark Blumenthal [xxx@xxx]date Thu, Feb 4, 2010 at 4:17 AMsubject Research 2000 weirdnessmailed-by gmail.comMark,Not to sound too conspiratorial, but to be honest I'm gettinga little bit suspicious about Research 2000, or at least thepolling they've conducted for Markos over the past two years.Do you know those guys at all?I'll keep this pretty brief.  In part it's because of theoccasionally really weird result they turn out -- for instance,they had only 27 percent of Republicans or something in favorof gays in the military whereas Gallup and ABC/Post have hadthose numbers in the 60s.  There are two or three other exampleslike this I could point to.  For another, their contactinformation and web presence is pretty sketchy relative to thatof other pollsters and there's not a lot of detail about thescope of their operations.But mainly, it's that that their data feels way too clean forme.  Take a look at the attached chart, for example: these arethe age breakdowns in the Democratic vote share for the last20 contests surveyed by R2K and PPP, respectively.  The agebreakdowns in Research 2000's numbers are almost always closeto "perfect" -- in 20 out of 20 cases, for instance, theDemocrat gets a lower vote share from among 30-44 year oldsthan among 18-29 year olds.  PPP's data, on the other hand,is *much* messier -- which is what I think we should expectwhen comparing small subsamples, particularly subsamples oflots of different races that are subject to differentdemographic patterns.Likewise, take a look at their Presidential tracking numbersfrom 2008 (http://www.dailykos.com/dailypoll/2008/11/4).They published their daily results in addition to theirthree-day rolling average ... and the daily results wereremarkably consistent from day to day.  At no point, forinstance, in the two months that they published daily resultsdid Obama's vote share fluctuate by more than a net of 2points from day to day (to reiterate, this is for the dailyresults (n=~360) and not the rolling average).  That justseems extremely unlikely -- there should be more noise thanthat.Maybe/probably they're just using some weighting proceduresthat smooth out a lot of the noise that you would ordinarilyexpect to see, but it all looks pretty weird to me.Anyway, let me know your thoughts.  If you think there'senough smoke there, my next step would probably be to bringthis to Markos's attention.NateAttachment:On Thu, Feb 4, 2010 at 4:46 AM, Nate Silver wrote:>> OK, here's something else weird.  This is the Democratic shareof the vote in the last 30 races that they've surveyed, this timebroken down by gender.  The Democrat does better among women inall 30 cases -- no doubt that's generally going to be the case, butbetween races local idosyncracies and sampling noise, I don't knowif the poll should hit the bullseye that often.  What's also weirdis that in every single case, the gender gap is an even number.

As I stipulated in the e-mails, this evidence was fairly circumstantial — and Mark suggested to me that the results could have been caused by excessive party weighting rather than anything more ulterior. So I declined to go public with them, although I did forward a copy of this information to Markos and encouraged him to do additional due diligence on Research 2000.

In retrospect, of course, I wish that I had been a bit more dogged about this. But I’m happy that Grebner, Weissman and Weissman — whom, to be clear, I was not in touch with during this process, and from what I understand initiated their research independently of Daily Kos — came along to study the issue properly. Their work is excellent and is to be commended.

Filed under

All Nate Silver

All Politics