Skip to main content
ABC News
How Not To Take Responsibility For a Failed Forecast

The House race forecasts issued by Stu Rothenberg and his team on the eve of last November’s elections performed extremely well.

Not only did the Rothenberg Political Report project the overall number of Republican gains correctly — “[a] likely Republican gain of 55-65 [House] seats, with gains at or above 70 seats possible,” they wrote (the Republicans actually gained 63 seats in the House), but the Rothenberg team also did quite well in projecting the outcome of individual races.

If you assess them one point for every missed won-loss prediction and half a point for each race in which they declined to designate a favorite and called the race a toss-up, then Mr. Rothenberg’s team scored just 18 points on the night (fewer points is better), compared with 18 for Larry Sabato’s forecasts; 19 for FiveThirtyEight; 29 for RealClearPolitics; 29.5 for CQ Politics; 31 for the Cook Political Report, and 33 for HuffPost Pollster (HuffPost Pollster did not issue forecasts in a few competitive races, so I assigned them the Cook Political forecast in these cases).

The night after the election, once I’d seen how well Rothenberg’s team had done, I sent a note to Nathan Gonzales, the political editor of the Rothenberg Political Report, complimenting the team on their performance. Their forecasts have been quite reliable in the past as well, which is why we incorporate them — along with forecasts from Cook Political, CQ Politics and Mr. Sabato — into our own House race projections.

Much earlier in the political cycle, however, Mr. Rothenberg had made a very silly forecast. This is what he wrote on April 23, 2009 (emphasis is mine):

Over the past couple of weeks, at least three Republicans — House Minority Whip Eric Cantor (Va.), former Speaker Newt Gingrich (Ga.) and campaign consultant Tony Marsh — have raised the possibility of the G.O.P. winning back the House of Representatives next year.

That idea is lunacy and ought to be put to rest immediately.

None of the three actually predicted that Republicans would gain the 40 seats that they need for a majority, but all three held out hope that that’s possible. It isn’t.

[…] There are no signs of a dramatic rebound for the party, and the chance of Republicans winning control of either chamber in the 2010 midterm elections is zero. Not “close to zero.” Not “slight” or “small.” Zero.

Mr. Rothenberg couldn’t have made his point more emphatically: Republican chances of taking over the House, he wrote, were not just slim but none.

Well, he got that one very, very wrong. It’s no huge deal: we all make mistakes. I’ve certainly made a few of them. When a Rasmussen Reports poll came out in Massachusetts this time last year, showing Scott Brown within 9 points of Martha Coakley, I said I thought there was only a 3 to 5 percent chance that Mr. Brown would win. And in August 2009, I said I thought the unemployment rate probably wouldn’t exceed 10 percent, which it eventually did.

I did hedge these forecasts somewhat. After the Rasmussen poll, I said that Mr. Brown’s chances, while not very strong, were credible enough that they ought to worry Democrats. And I said that while I expected the unemployment rate to remain below 10 percent, there was a 1-in-3 chance that it would exceed it.

I go to great lengths to do this — to represent my forecasts in terms of probabilities, not certainties — because the world is a complicated place and there’s only so much we can really know about it; few things are truly impossible, especially in the world of politics. (Okay, the Republicans aren’t going to introduce a cap-and-trade bill tomorrow — that is a lock.)

It can sometimes be tempting, when one looks at the world probabilistically, to excuse any incorrect forecast by saying you have merely been “unlucky.” After all, even if you are correct, based on the information available to you at the time, that some event has only a 5 percent chance of occurring, sometimes the dice will come up snake eyes, and the long shot will come through.

Of course, it is easy to abuse this privilege. So the question is how well calibrated your forecasts are in the long run. If out of all of the events that you say have a 5 percent chance of occurring, 20 percent of them eventually do happen, you’ve got a problem. But if none of the events that you say have a 5 percent chance of happening ever occur, you’ve also got a problem.

Mr. Rothenberg did not build in any hedge, however, so he has to fall back on other excuses, a couple of which he offers in a new article at Roll Call.

One of them is semantic: he was making a “projection,” he says, rather than a “prediction.”

“I’ve always seen what I do as handicapping, not predicting,” Mr. Rothenberg writes. “For as we all know, long shots win in both horse racing and politics.”

Mr. Rothenberg is right that there is a difference between a forecast and a prediction, even though the terms are often used interchangeably in everyday language. It will snow tomorrow — this is a prediction; there is a 70 percent chance that it will snow — that is a forecast.

Mr. Rothenberg also sometimes uses a third term, projection, in his article, which should probably be avoided when one is trying to draw fine points of distinction since it has a vaguer meaning than the other two. With that said, his meaning is clear enough and his attitude is just the right one: a responsible forecaster sets the odds, rather than issuing categorical predictions about what will or will not happen.

When you say that something has a zero percent chance of happening, however — particularly when you go to great lengths to say that the chance is exactly zero and not almost zero, as Mr. Rothenberg did in handicapping the chances that the Republicans would take over the House — this distinction between forecasting and prediction becomes meaningless.

Notwithstanding this somewhat esoteric point — let’s assume that, when Mr. Rothenberg said “zero”, what he really meant was “very, very low” — he offers a few more substantive arguments. For instance, he writes:

As I wrote in that column, the G.O.P. needed a political wave to develop for that to happen, and in April 2009 there were no signs of a wave developing. None. Zero. Zilch. In fact, at that time, public sentiment about the future (specifically about the direction of the country) was improving.

Is it true that, as of April 2009, there were no signs of a Republican wave developing?

No, it isn’t. There were definitely some signs that the political climate was shifting. For instance, by April 23, 2009, the Republicans had already narrowed the Democrats’ advantage on the generic congressional ballot from 7 or 8 points to about 2, which clearly suggested that the House was in play:

Some polls, in fact, already had Republicans ahead on the generic ballot. Noting this on March 23, 2009, I said that I thought the Republicans still had a long way to go — but that political futures markets, which were then pegging the Republican odds of a takeover of the House at 20 percent, were probably too low.

In addition, April 2009 was a month of Tea Party protests; Tax Day rallies drew at least 250,000 people in hundreds of locations across the country. I can’t claim to have had any great insight at the time about whether the Tea Party protests were straws on a wind of much broader dissatisfaction throughout the electorate, or whether they were an isolated event. But clearly, there were some tangible signs of a shift against Mr. Obama and the Democrats, however much meaning one ultimately wanted to assign them. Also, by summer 2009 (although perhaps not as soon as April) there was also some evidence from polling that Republican voters were significantly more excited about the election than Democrats: an “enthusiasm gap” was developing.

Even if one chose to ignore these pieces of evidence, and concluded that the political climate was still strong for Democrats at that time, there were reasons to believe that it might shift before the following November. The unemployment numbers released in January, February, March and April that year were awful, and the I.M.F. was already forecasting that unemployment would remain at 9 percent throughout 2010. A lot of people — myself included to some extent — had some trouble putting two and two together, but it ought not to have been so difficult to recognize that this could eventually cause great problems for the Democrats, as voters assigned Mr. Obama progressively more blame for the condition of the economy — especially given the ambitious and expensive agenda he and his party were pursing.

Finally, there is some evidence from political science that the political climate usually tends to shift against the incumbent party in advance of a midterm election. One should have built in a hedge for that reason alone, as well as for the fact that electoral forecasts 18 months in advance are almost always subject to a huge amount of uncertainty.

Granted, not very many people fully grasped the magnitude of the Democrats’ problems as early as April 2009, Mr. Obama’s fourth month in office. About the only exceptions I can think of are Paul Krugman, who thought the stimulus package was too small to yield a robust economic recovery and that Democrats would bear the political burden of this, and Jay Cost — then of Real Clear Politics — who noted that the country was still extremely polarized and that there was significant downside risk for the Democrats, particularly if they pursued an aggressive agenda that the public might not have an appetite for.

Neither Mr. Krugman nor Mr. Cost was explicitly saying that the Democrats were more likely than not to lose the House, as far as I can recall. Still, it was certainly a distinct possibility based on the evidence available at the time, and few observers went as far as Mr. Rothenberg in dismissing it out of hand.

Keep in mind that Mr. Rothenberg’s strength — and it’s the same strength that I have, really — is in looking at politics at a microscopic scale. For instance, he and his team are very good at weighing the evidence — polls, candidate quality, insider gossip, demographics — and coming to a forecast of how likely the Democrat or Republican is to win in each state or congressional district. This is hard to do, especially in races for the House, when much of the evidence demands a skeptical eye — the polling in such races is often dubious, for instance.

There is another category of political forecasting that one might call macroscopic: looking at broad, sweeping trends, and coming to some conclusions about how the political climate might change over the long term. This is probably even harder, and I can think of almost nobody who really has an aptitude for it. On the contrary, people usually make the mistake of extrapolating either the status quo or the most recent result (Obama just won = permanent Democratic majority!!!!) into the indefinite future.

Mr. Rothenberg is a “micro” guy who tried to go “macro” and made a mistake. That’s okay; few people do the micro as well as he does, and almost nobody gets the macro stuff right anyway. Still, it’s a mistake that Mr. Rothenberg ought to be more willing to own up to.

Nate Silver founded and was the editor in chief of FiveThirtyEight.