Skip to main content
ABC News
Oct. 21: Uncertainty Clouds Polling, but Obama Remains Electoral College Favorite

The bad news for President Obama: it’s been almost a week since the second presidential debate, in Hempstead, N.Y., one that instant-reaction polls said was a narrow victory for him. But there is little sign that this has translated into a bounce for Mr. Obama in his head-to-head polls against Mitt Romney. Instead, the presidential race may have settled into a period of relative stability.

There is bad news for Mr. Romney as well, however. The “new normal” of the presidential campaign is considerably more favorable for him than the environment before the first debate, in Denver. However, it is one in which he still seems to be trailing, by perhaps 2 percentage points, in the states that are most vital in the Electoral College.

The FiveThirtyEight forecast was essentially unchanged again on Sunday, with Mr. Obama retaining a 67.6 percent chance of winning the Electoral College, little different from his 67.9 percent odds on Friday and Saturday.

These estimates might seem to be incongruous with national polls that show a nearly tied race. But the FiveThirtyEight method is, principally, an Electoral College simulation, and therefore relies more heavily on state-by-state polls. Our estimates of the popular vote in the critical states are highly similar to those of other Web sites that use different methods to calculate them.

This is expressed in the chart below, which compares the current FiveThirtyEight “now-cast” in each battleground state (our snapshot of what would happen in an election held today) against those of three of our competitors, RealClearPolitics, HuffPost Pollster and Talking Points Memo.

There are only three states that are “called” differently by the different methods. In Colorado, RealClearPolitics gives a slight edge to Mr. Romney, while the other three sites have Mr. Obama just ahead. However, it would be easy to overestimate the importance of these differences. RealClearPolitics has Mr. Romney only 0.2 percentage points ahead there, while the most favorable result for Mr. Obama, from The Huffington Post, has him up by 1.3 percentage points.

In Virginia, RealClearPolitics and The Huffington Post show an exact tie, while FiveThirtyEight and Talking Points Memo give a slight advantage to Mr. Romney.

Finally, in New Hampshire, FiveThirtyEight and The Huffington Post have Mr. Obama ahead, while the other two sites have the race going for Mr. Romney.

Still, these states — Virginia, Colorado and New Hampshire — are not quite at the electoral tipping point. Instead, Mr. Obama could win the Electoral College by winning Ohio, Wisconsin, and either Iowa or Nevada.

All the methods have Mr. Obama just ahead in each of these states. In Iowa, his margin runs from 1.5 percentage points to 2.5 percentage points, according to the different methods, with FiveThirtyEight representing the lowest point in the consensus. In Nevada, it is between 2.1 points and 5.2 points, with FiveThirtyEight again reflecting the lowest end of the range.

FiveThirtyEight is somewhere in the middle of the pack in Ohio, where the different methods give Mr. Obama an advantage of between 1.4 and 2.6 points, and in Wisconsin, where the estimates run between a 2.8- and a 3.8-point advantage.

One can certainly debate how to translate Mr. Obama’s apparent edge in each state into his overall probability of winning the Electoral College. It is a challenging problem from a mathematical standpoint, particularly given that the states cannot be expected to behave independently from one another. (That is, if Mr. Obama underperforms his polls in one swing state like Ohio, he is also more likely to do so in similar ones like Iowa.) It is a problem that we thought about very carefully in designing the model this spring, and in advance of the 2008 election. But because there have been only about eight presidential elections in which there has been a rich amount of state-by-state polling data, any approach will inherently be speculative to some degree.

At the same time, off-the-cuff or “gut-feel” estimates of these probabilities are probably not a good substitute for an effort to determine them through a more rigorous, empirical and rule-driven means. Our intuitive estimates of probabilities are normally quite poor, especially in situations that involve a relatively large degree of mathematical complexity and in which we have relatively little hands-on experience.

To be sure, these problems also present difficulties for anyone seeking to design a forecast model.

But if you accept the premise that Mr. Obama is ahead by some (small) margin in the tipping-point states, something that all the different methods agree on, it then becomes a question of how much doubt you should have about that advantage given the intrinsic uncertainty in polling.

Saying that the race “could go either way” is an obviously correct statement — but also one devoid of insight.

We dare to pose a more difficult question, the one that a gambler or an investor might naturally ask: What are the odds?

We calculate Mr. Obama’s odds as being about two chances out of three.

Not only will the underdog — Mr. Romney — win some of the time, but he should win some of the time if we have estimated the odds correctly. If the set of candidates you have listed as 67 percent favorites in fact win 95 percent of the time, or 100 percent of the time, you’ve done something wrong. Over the long run, such candidates should win two out of three times — no less and no more.

Of course, it takes a very long time to realize the long run in presidential elections, since there is only one of them every four years. To the extent that one is evaluating the accuracy of political forecast models — whether they calibrate the odds correctly — it is probably better to look at something like races for the Senate. In that case, there are roughly 35 races held every other year, as opposed to just one every four years. Although these races are not completely independent from one another (there have been years in which Democrats or Republicans overperformed their Senate polls across the board), they are substantially more informative on balance for measuring the effectiveness of a series of forecasts.

Having indulged myself and imposed on your patience with this philosophical detour, let me turn your attention back to the presidential race.

Sunday’s Polls

Four of the five national tracking polls published on Sunday showed a gain for Mr. Romney, but there was one clear exception: a poll for Investor’s Business Daily, which showed Mr. Obama surging to a 5.7-percentage
-point lead from 2.6 percentage points previously.

What to make of the Investor’s Business Daily poll? Probably about the same thing that you make of the Gallup poll, which had Mr. Romney up by 7 points among likely voters, and had him gaining an additional point on Sunday.

You should be highly skeptical that these polls represent the true condition of the race: polls that look like outliers normally prove to be so. This is certainly the case with the Gallup poll, which has performed quite poorly in the past when it has diverged from the consensus of other surveys.

But the Investor’s Business Daily poll, like the Gallup poll, also has a history of sharp swings to and fro (sometimes before conveniently settling in with the consensus at the very end of the race).

When I point out that polls like Gallup or Investor’s Business Daily have a history of behaving in this way, my goal is not to browbeat them into changing their methods. Instead, as strange as this might sound, I would much rather that the pollsters paid no attention to FiveThirtyEight. When pollsters feel as though they are under pressure to conform to expectations about the race, they may herd toward the poll averages, which would reduce their independence and would therefore reduce the benefit of aggregating different polls together.

My intention, rather, is to provide a counterweight to what seem like misinformed views about the presidential race. Sometimes the outlying polls receive more attention from the news media, when they probably deserve less.

On Sunday afternoon, I looked up how often each tracking poll had been cited in the Lexis-Nexis news database over the course of the past week. (The criterion I used for this search was to look for instances in which the pollster’s name appeared within 25 words of the term “tracking” and also either of the terms “poll” or “survey.”)

It turns out that the Gallup national tracking poll was cited in the news media more often than the other six national tracking polls combined. Although the national tracking polls show, on average, a race that is about tied, they would have conveyed the impression of a four-point lead for Mr. Romney if weighted based on how often they were cited in news accounts.

The simplest and best defense against this is merely to take the average of the polls.

Even averaging methods can conceal a fair amount of intrinsic uncertainty, however.

The nine national polls included in the RealClearPolitics average as of Sunday evening, for instance, contained about 12,000 interviews between them. (Collectively, they showed Mr. Obama ahead by a trivial margin of 0.2 percentage points.)

The margin of sampling error on a 12,000-person sample is larger than you might think: about plus or minus 2 percentage points in measuring the difference between the two major candidates. So the national polls reflected in the RealClearPolitics average could in fact be consistent with anything from a two-point lead for Mr. Romney to one of about the same margin for Mr. Obama.

And that calculation is based on just one type of error, that which is unavoidable from taking a random subsample of a larger population.

In fact, there are many other types of ambiguities in polling. Whether polling firms include cellphones in their samples seems to make a difference of about 2 percentage points, for instance. (The firms that include cellphones normally show stronger results for Mr. Obama.)

The likely-voter models that polls employ vary substantially from one another, suggesting everything from a situation in which Mr. Romney is performing about one point better among likely voters than among the broader universe of registered voters to one in which he has as much as a six- or seven-point advantage instead.

Some demographic groups are harder to get on the phone than others, and the techniques used to correct for this vary in quality. Polls that are especially “bouncy” may use poorly designed demographic weighting algorithms.

These same issues also apply in polls conducted at the state level, of course.

There are three major defenses that the FiveThirtyEight model employs in recognition of these problems.

First, some of the “noise” in polling is predictable, since some polling firms consistently show results that are more favorable for one or the other candidate. In the cases in which the pattern is most consistent, we correct for this by means of our house effects adjustment.

Second, we seek to use state polling data and national polling data in a holistic way. It might seem, if you are looking at the state polls and the national polls in a very literal-minded way, that Mr. Obama is quite likely to win the Electoral College despite losing the popular vote. We certainly do allow for some possibility of this, and it has tended to increase over the course of the year. (Our estimate is that the chances would be about 7 percent in an election held today, although there is also a 3 percent chance of the reverse occurring and Mr. Romney winning the Electoral College despite losing the popular vote.)

To regard the probability as being substantially higher than that — taking the RealClearPolitics state and national polling averages might imply a 20 percent probability of a split outcome, for instance — may be to overestimate the degree of precision in both state and national polling averages.

What appears to be a tied race in national polls, as I mentioned, could fairly easily be a two-point lead for Mr. Obama just based on statistical variance alone, in which case there would be no appreciable difference between the national polls and the polls in the tipping-point states. By the same calculation, of course, the national polls could also be consistent with a two-point lead for Mr. Romney.

But our finding is that both state and national polls provide useful information about the overall condition of the race. The state polls can be aggregated, and weighted based on each state’s projected turnout, to provide an estimate of the national popular vote. In the past, this technique would have provided an estimate of the national popul
ar vote that would sometimes have been more accurate than the one from national polls themselves.

Mr. Obama’s swing state polls are consistent with the hypothesis that he holds some sort of Electoral College advantage, but they are also consistent with the hypothesis that the national polls have the race slightly wrong. If you start out with the premise that the national polls are unimpeachably precise, then you might not give much weight to this hypothesis. But they are not all that precise as a matter of either theory or practice.

The opposite hypothesis — that the state polls are overestimating Mr. Obama’s standing — is also a valid one to consider. If you return to the first chart in this article, you’ll find that the FiveThirtyEight “now-cast” provides the most optimistic estimate for Mr. Obama’s standing in the national popular vote (it puts him ahead by 0.8 percentage points). But we are generally on the lower end of the consensus on Mr. Obama’s standing in swing states like Ohio, Iowa and Nevada. This is because, when the model detects an overall discrepancy between state and national polls, it adjusts both of them in opposite directions to bring them closer to one another. If after that adjustment the model still finds evidence that one candidate is performing especially well in the most important states, it will then attribute the remainder of the difference to a potential popular vote and Electoral College split.

Following a similar theme, we also use both national and state polls to inform our view about the overall trend in the race. The state polls published over the past several days, including on Sunday, have shown some stronger and weaker numbers for the candidates, but they are generally consistent with a race that has not changed much since Mr. Romney made his gains after the Denver debate.

Be wary of analyses that suggest that one candidate is surging or declining by an especially large amount in one or two particular states in a way that diverges from the overall trend in the race. Many of these apparent changes merely reflect statistical noise. We are at the point, certainly, where if Ohio moves a little less than the national polls, or Florida moves a little more, that is potentially significant. But most people’s first instinct is to attribute too much meaning to these changes.

That brings up the third line of defense in the FiveThirtyEight model: our estimate of the uncertainty in the calculation. These estimates are based on how well polling averages, and presidential forecasting models, have performed under real-world conditions, rather than under idealized assumptions that, for instance, the uncertainty in polls is limited to statistical sampling error alone, or that the vote in each state is independent of the next one.

Right now we’re at a point where, for better or for worse, these methodological choices will make quite a lot of difference in your evaluation of the race. The best I can do is to explain why we made the choices that we did in designing the model, and how they are playing out given the information that is coming our way.

There remains an outside chance that the race will break clearly toward one or the other candidate, after the third debate on Monday or because of some intervening news event, but the odds are strong that we will wake up on Nov. 6 with a reasonable degree of doubt about the winner. For that matter, we may wake up on Nov. 7 still uncertain about who won.

Nonetheless, stipulating that the race is clearly very close is not an adequate substitute for placing any kinds of odds on it at all. And the central premise behind why we see Mr. Obama as the modest favorite is very simple: he seems to hold a slight advantage right now in enough states to carry 270 electoral votes.

Nate Silver is the founder and editor in chief of FiveThirtyEight.


Filed under