Nate Silver has criticized election forecasting models before, but his latest post is perhaps the most thorough critique. I am not an elections forecaster, but I am a political scientist and consumer of these models. Nate has kindly let me abuse my privilege as an occasional guest blogger here at FiveThirtyEight to respond to his critique.
Part of Nate’s critique has to do how these models are sometimes (though not universally) constructed and modified. For example, he criticizes ad hoc adjustments to models to account for idiosyncratic features of a single election — fitting the model to noise rather than signal, as it were. I agree. He has also noted that elections forecasts have a lot of uncertainty and that more could be done to emphasize this. I agree with that, too. In fact, elections forecasters often raise similar points by way of critiquing of each other.
But I am less critical of the accuracy of these models than is Nate. For one, forecasters have different motives in constructing these models. Some are interested in the perfect forecast, a goal that may create incentives to make ad hoc adjustments to the model. Others are more interested in theory testing — that is, seeing how well election results conform to political science theories about the effects of the economy and other “fundamentals.” Models grounded in theory won’t be (or at least shouldn’t be) adjusted ad hoc. If so, then their out-of-sample predictions could prove less accurate, on average, but perfect prediction wasn’t the goal to begin with. I haven’t talked with each forecaster individually, so I do not know what each one’s goals are. I am just suggesting that, for scholars, the agenda is sometimes broader than simple forecasting.
Second, as Nate acknowledges but doesn’t fully explore (at least not in this post), the models vary in their accuracy. The average error in predicting the two-party vote is 4.6 points for Ray Fair’s model, but only 1.72 points for Alan Abramowitz’s model. In other words, some appear better than others — and we should be careful not to condemn the entire enterprise because some models are more inaccurate.
Third, if we look at the models in a different way, they arguably do a good enough job. Say that you just want to know who is going to win the presidential election, not whether this candidate will get 51 percent or 52 percent of the vote. Of the 58 separate predictions that Nate tabulates, 85 percent of them correctly identified the winner — even though most forecasts were made two months or more before the election and even though few of these forecasts actually incorporated trial heat polls from the campaign.
This view reflects my “forest, not the trees” approach to consuming these models. I assume that any individual model will always have errors. I assume that although some forecasters are historically more accurate than others, no one has some special forecasting sauce that makes his model the best. So when I see a range of forecasts, I tend to look at the direction that forecast is pointing. That tells me who is likely to win. Looked at this way, the “forest” will rarely lead me astray in “Dewey Defeats Truman” fashion. Perhaps that’s a low bar, but that’s all I am looking for. (And, as Election Day draws closer, there will always be purely poll-based forecasts to draw on as well, both nationally and within states.)
To be sure, the forest-not-trees approach does not render criticisms of forecasting models irrelevant. Moreover, forecasters themselves often use “the trees” — i.e., errors in any one model’s predictions — to evaluate the models. So Nate is entirely justified in using these metrics himself. I am also not suggesting that problems in forecast models should be ignored as long as they get the winner right — after all, some models called the winner correctly but overestimated his vote share by 10 points — or that the models cannot be improved, or that there might be better ways of forecasting elections than any of these models. I am simply suggesting that viewed at a distance, the models will rarely “fail” (as the headline of Nate’s post has it) in a way that misleads the average person who follows politics and wants to know only who’s the likely winner, but doesn’t care about root-mean-square error.
Let me conclude by amplifying and extending one point that Nate makes, which gets at the implications of problems in election forecasting models. He notes that problems in forecasting models do not mean that the economy doesn’t influence elections, and that political science research on campaigns and elections goes far beyond forecasting models. That’s an important point. For example, there has been a real renaissance in the study of campaign effects in political science over the last 20 or so years. (I say this as someone who wrote a dissertation on the subject.) There have been ambitious data-gathering projects — of television ads and surveys, for example. There have been notable books on presidential campaigns — by Lynn Vavreck, Tom Holbrook and Daron Shaw and by Richard Johnston, Michael Hagen, and Kathleen Hall Jamieson, among others — as well as many journal articles (here is one example). Not all of this work shows big campaign effects or any effects, for that matter. But a significant amount of research looks at when and how campaigns matter. Ms. Vavreck and I are writing a book on the 2012 election precisely to investigate this.
Given this research, we should not — we don’t have to — interpret the errors or shortcomings of forecasting models as de facto evidence that aspects of candidates or campaigns matter. (Which is something Nate is careful not to do as well.) As I’ve previously noted at the Monkey Cage, there is a tendency to put influences on elections into two categories: the economy/fundamentals and candidates/campaigns/messages/gaffes/etc. So to hear that change in gross domestic product predicts only about 40 percent of the variation in presidential election outcomes could lead someone to say “Wow, the campaign explains the other 60 percent!”
But you can’t disentangle the impact of the economy and the campaign that easily. The characteristics of candidates and campaign strategies themselves depend on the economy. Better candidates will challenge incumbents when the economy makes those incumbents vulnerable. Candidates’ decisions to campaign on the economy will depend on whether they will get credit or take blame for economic conditions. And factors like the economy often come to matter precisely because they are emphasized in the campaign. That is to say, the campaign can “make” the forecasting models come true, or at least truer.
In short, the economy is bound up with campaigns and elections in may complex ways. There is no simple way to separate the total effects of structural forces like the economy and the total effect of the campaign itself. So although I’ll continue to follow forecasting models and hope that Nate’s comments, among others, make those models better, the bulk of what we can learn and should understand about elections will not come from forecasting models.