There was a lot of discussion Monday about the rebuttal by the Democratic Senatorial Campaign Committee (DSCC) to FiveThirtyEight’s latest Senate forecast. That forecast identified the Republicans as more likely than not to take over the Senate. But the GOP’s advantage is slim, and there’s a lot of uncertainty in both the individual races and the overall forecast.
The DSCC’s memo pointed to past forecasts by FiveThirtyEight that were off the mark. It’s a fun story for news outlets, but public statements by partisan groups won’t usually say anything that you didn’t already know.
Indeed, it’s not news that forecasts are sometimes wrong — as our Senate forecasts were in Montana and North Dakota in 2012. (Democrats won both races when Republicans were favored in our model.) Furthermore, the margin of error is larger at earlier stages of the campaign.
That’s why our forecasts are expressed in terms of probabilities. For example, our NCAA tournament model gave Mercer just a 7 percent chance of defeating Duke on Friday. Mercer won. Upsets are supposed to happen sometimes. Specifically, out of all forecasts in which we say the underdog has a 7 percent chance of prevailing, the underdog is supposed to win about seven times out of 100 over the long run — no more and no less. This property is called calibration, and it’s one of the best ways to assess probabilistic forecasts. (We’ll be conducting a test of the historical calibration of our NCAA forecasts this week.)
Nor is it news when party officials claim that the errors are all in their direction. During the 2012 presidential campaign, Republicans were convinced that Mitt Romney would significantly outperform the polling averages. In fact, the polls weren’t all that great in 2012. But the polls were biased (I use the term “bias” strictly in a statistical sense) in favor of Romney, rather than against him. For instance, President Obama won Colorado by 5.4 percentage points as compared with a RealClearPolitics polling average that showed him ahead by 1.5 points; he won Virginia by 3.9 points as compared with a RealClearPolitics polling average of 0.3 points.
Sometimes, as in North Dakota in 2012, party officials are right and public polls are wrong. But more often, it’s the other way around, such as when the Romney campaign’s internal polls were badly biased toward its candidate, or when Democratic polls claimed the Wisconsin gubernatorial recall in June 2012 was a toss-up. (Instead, Scott Walker, the Republican governor, was retained by 7 percentage points — almost exactly what the public polls predicted.) On average, partisan polls released to the public express a 6-point bias in favor of their candidate.
Our forecasts could be wrong in November. In fact, they probably will be wrong — it’s unlikely that Republicans will win exactly six seats. But we think it’s equally likely that our forecast will be biased in either direction. If Democrats retain just one more seat, they’ll hold the Senate. Or Republican gains could grow to seven seats, or quite a bit more.
And here’s the least surprising news: Political campaigns are hypocritical. At the same time the DSCC is criticizing our forecasts publicly, it’s sending out email pitches that cite Nate Silver’s “shocking, scary” forecasts to compel Democrats into donating.
You’d do well to shut out the noise the next time the DSCC writes a polling memo.