Charles Franklin has a typically articulate analysis up at Pollster.com which comes to a somewhat different conclusion than mine about the state of play in Massachusetts. His analysis works by lumping polls together into different bundles (e.g. “non-partisan”, “Republican + non-partisan”) and producing the following graph:
This certainly looks persuasive. But I’m not sure if it’s as robust as it appears. What happens, for instance, if instead of bundling together different types of polls, we instead let each poll speak for itself?
Clearly, Scott Brown is in a much better position than he was two weeks ago, and probably in a better position than he was one week ago. But otherwise the story is somewhat more ambiguous. There are two pollsters, Coakley and Pajamas, that in fact show some kind of movement back toward Coakley in the last couple of days (note that I’m taking advantage of one data point that Franklin didn’t have, which is that Coakley is supposedly now up by 2 in her own internals, according to the Boston Herald).
Yes, in Pajamas’ case it’s from a -15 to a -9 and is possibly just reversion to the mean; and no, I’m not entirely comfortable with taking Coakley’s internals at face value (that holds for the Pajamas numbers too). But keeping those caveats in mind, it’s not that difficult to draw an übertrendline that is roughly consistent with the story told by each of the individual trendlines. It would probably look something like this:
This would have Coakley losing ground fairly steadily through about the 14th, and then perhaps having added back 2 or 3 points since then. It does a good job of staying parallel with all four trendlines formed by the individual pollsters. Note, for instance, that the decline in Coakley’s PPP numbers actually wasn’t that significant (-4 points) considering that there was a 9-day gap between their polls; if you buy the übertrendline’s story, this may be because they didn’t poll at Coakley’s bottom.
The übertrendline is less successful at tracking the red diamonds, which are the orphans — the pollsters that have only surveyed the race once. But, for varying reasons — turnout is hard to model in special elections; a lot of these pollsters are dodgy; some of them are partisan — we have seen very large house effects in the polls of this race and some of what looks like a trend may just be different pollsters with different slants on the race happening to have weighed in at different times. What’s unusual is that, if you took the orphan polls and drew a trendline through them, it would be about twice as steep as the übertrendline, which a composite of actual trendlines. In other words, those orphan polls make Franklin’s trendline both appear both steeper and more linear than it perhaps actually is.
The concept of the übertrendline — the trendline of trendlines — is pretty much exactly what our Presidential election model attempts to calculate. What it’s trying to discern, essentially, is this: if every pollster surveyed every state every day, what would the average look like? If, for example, the Boston Globe, which previously had the race at Coakley +17, weighed in on the race this morning, what we expect it to show? Surely not Coakley +17 again — you’d have to subtract some points for the übertrendline and possibly also for reversion to the mean. But I’m guessing that it would probably show Coakley ahead, and possibly outside the margin of error. Conversely, if Pajamas, which has a huge house effect in the other direction, had polled the race back on January 4th when the Globe did, what might they have found then? Perhaps something close to a dead heat, I’m guessing.
Guessing, guessing — I’m using that word a lot, guessing. When we’re looking at a Presidential race, we don’t have to guess as much, because we can look not only at polls in a particular state, but also at polls from other, similar states, as well as national polls, including several daily trackers. Under those circumstances, this method can be quite robust. Here, it is less so.
But this is also true for the Pollster.com approach. If you take the Pajamas Media polls out of their average, for instance, their characterization of the race goes from Brown +7.9 to Coakley +1.2! That’s not especially robust, either.
This race and NY-23 have been exceptionally challenging to polling analysts; I unapologetically hedged about as much as I could in NY-23, and that’s my attitude again here. There is definitely more than one storyline that you can tell about the polling in this race, as well as more complicated storylines that involve figuring out in which direction the polls might be off. That is why I continue to characterize the race as a toss-up. It’s not necessarily that I’m predicting a super close finish or recount or anything like that — there’s certainly a decent possibility of that, but I I could envision a double-digit win for either candidate. It’s more that our point estimates of where the race stands is swamped by the uncertainty.
As the remaining polls come in, you should image how they might impact the übertrendline. If the Rasmussen poll comes in somewhere between Brown +1 and Brown +3, that would tend to validate that the übertrendline is correct. If they come in at Brown +6 or something, that would suggest that the any late uptick in Coakley’s numbers is probably illusory, and something closer to Franklin’s impression of a more linear decline may be correct. And if they hold at Coakley +2, that would obviously be a good data point in her favor.