Yesterday, I had a telephone conversation with John McIntyre of Real Clear Politics about their policies for including or disincluding certain polls in their national averages. In particular, I was curious as to why they do not include the Research 2000 daily tracking poll, which is conducted on behalf of Daily Kos.
McIntyre’s position is as follows. There are too many polls, and particularly too many national tracking polls (on this point, I tend to agree). They decided that they had to draw a line in the sand, and will not be including any more “new” tracking polls that do not have a history of polling the national horse race. This would include the Research 2000 poll, but also things like Insider Advantage, which put out some national numbers a couple of weeks ago, or the Big Ten poll released last week, which also included a national trial heat.
McIntyre said that he thinks Del Ali (Research 2000’s CEO) is a good pollster and that the decision to exclude them was difficult. He said he’d have made the same decision if an organization like SurveyUSA or Mason-Dixon, which have not traditionally done a lot of national polling, began their own tracking poll. He also said that he will likely include R2K in 2012.
Naturally, I had a couple of questions for John. In fact, I gave him a pretty hard time. Why include something like the Associated Press-GfK poll, as RCP did a couple of weeks ago? That polling firm has no track record whatsoever. McIntyre said that he’s willing to make an exception is a major media organization like the Associated Press puts its backing behind a poll. He doesn’t consider Daily Kos a major media organization, although that has nothing to do with their partisan affiliation.
Why use Research 2000’s state-level numbers — as RCP does, including the state polls that are commissioned by Daily Kos — but throw out their national data? Because, McIntyre says, state polling is harder to come by — and so in essence they’re more willing to take what they can get.
*-*
RCP’s position is logically consistent. It’s not just the R2K poll they’re excluding — there are lots of national polls that they don’t list. And it is absolutely the case that you may run into some difficulties if you open up the tent and are willing to list any and all polls. Some polls simply aren’t any good, and at the extreme case, you run into the circumstance where someone simply puts together a PDF in his basement and blasts it out to the various polling aggregators. Are we supposed to include such polls or not?
Here at FiveThirtyEight, we have a couple of advantages that allow us to be fairly liberal about which polls we include. In fact, we include all polls, unless they are conducted on behalf of a campaign, Political Action Committee (this would include something like a union), or a national party. Firstly, we weight polls based on their historic track record, and assign any “new” polls a below-average weight. And secondly, we correct for house effects. So if someone decided to commission a new poll and juice it 5 points for the Democrats (or the Republicans), we’d just pull those 5 points right back out.
RCP does not do these things and is not particularly interested in doing them, so they instead need to make some judgment calls about which polls to include. It should be emphasized that RCP’s is not an all-inclusive collection of polls. It is a select list of polls, polls which pass their smell test and which they think are worthy of public consumption.
The critique I have — and I expressed as much to McIntyre — is that that if you’re going to draw a line somewhere, Research 2000 probably ought to be on the included side of it. They haven’t done much national polling before, but they have a strong track record at the state level, and national polling is inherently easier than state polling. And while they might not be associated with a national news outlet (a dubious criterion to begin with, since a lot of the major news organization polls aren’t all that good) they have been entrusted for years by dozens of newspapers like the St. Louis Post Dispatch at the state and regional level.
Ultimately, RCP’s approach and mine are a reflection of our respective business models. They are aiming for simplicity and ease of use, and trustworthiness. And they do these things well: RCP remains one of the first sites that I read every day. We are trying to take a more meticulous and proprietary approach — trying to provide a bit more alpha — but one which sometimes can sacrifice a pound of nuance for an ounce of accuracy.