Although I expect to proceed fairly carefully with respect to Research 2000, which Daily Kos will be suing for alleged fraud, I have suggested here and to at least one reporter that I had my own suspicions about Research 2000 which paralleled some of the findings in the study by Mark Grebner, Michael Weissman, and Jonathan Weissman. I want to be a bit more explicit about what I mean by that.
This is a copy of two e-mails that I sent to Mark Blumenthal of Pollster.com in the wee hours of the morning on February 4th. Like the examples in the Grebner study, they point toward cases in which Research 2000’s data appeared to be other-than-random (although, as I declaim in the e-mails, not necessarily triggered by fraud).
from Nate Silver [xxx@xxx]
to Mark Blumenthal [xxx@xxx]
Mark Blumenthal [xxx@xxx]
date Thu, Feb 4, 2010 at 4:17 AM
subject Research 2000 weirdness
Not to sound too conspiratorial, but to be honest I'm getting
a little bit suspicious about Research 2000, or at least the
polling they've conducted for Markos over the past two years.
Do you know those guys at all?
I'll keep this pretty brief. In part it's because of the
occasionally really weird result they turn out -- for instance,
they had only 27 percent of Republicans or something in favor
of gays in the military whereas Gallup and ABC/Post have had
those numbers in the 60s. There are two or three other examples
like this I could point to. For another, their contact
information and web presence is pretty sketchy relative to that
of other pollsters and there's not a lot of detail about the
scope of their operations.
But mainly, it's that that their data feels way too clean for
me. Take a look at the attached chart, for example: these are
the age breakdowns in the Democratic vote share for the last
20 contests surveyed by R2K and PPP, respectively. The age
breakdowns in Research 2000's numbers are almost always close
to "perfect" -- in 20 out of 20 cases, for instance, the
Democrat gets a lower vote share from among 30-44 year olds
than among 18-29 year olds. PPP's data, on the other hand,
is *much* messier -- which is what I think we should expect
when comparing small subsamples, particularly subsamples of
lots of different races that are subject to different
Likewise, take a look at their Presidential tracking numbers
from 2008 (http://www.dailykos.com/dailypoll/2008/11/4).
They published their daily results in addition to their
three-day rolling average ... and the daily results were
remarkably consistent from day to day. At no point, for
instance, in the two months that they published daily results
did Obama's vote share fluctuate by more than a net of 2
points from day to day (to reiterate, this is for the daily
results (n=~360) and not the rolling average). That just
seems extremely unlikely -- there should be more noise than
Maybe/probably they're just using some weighting procedures
that smooth out a lot of the noise that you would ordinarily
expect to see, but it all looks pretty weird to me.
Anyway, let me know your thoughts. If you think there's
enough smoke there, my next step would probably be to bring
this to Markos's attention.
On Thu, Feb 4, 2010 at 4:46 AM, Nate Silver wrote:
> OK, here's something else weird. This is the Democratic share
of the vote in the last 30 races that they've surveyed, this time
broken down by gender. The Democrat does better among women in
all 30 cases -- no doubt that's generally going to be the case, but
between races local idosyncracies and sampling noise, I don't know
if the poll should hit the bullseye that often. What's also weird
is that in every single case, the gender gap is an even number.
As I stipulated in the e-mails, this evidence was fairly circumstantial — and Mark suggested to me that the results could have been caused by excessive party weighting rather than anything more ulterior. So I declined to go public with them, although I did forward a copy of this information to Markos and encouraged him to do additional due diligence on Research 2000.
In retrospect, of course, I wish that I had been a bit more dogged about this. But I’m happy that Grebner, Weissman and Weissman — whom, to be clear, I was not in touch with during this process, and from what I understand initiated their research independently of Daily Kos — came along to study the issue properly. Their work is excellent and is to be commended.