Skip to main content
Menu
Outlier Polls Are No Substitute for News

On Wednesday morning, Bloomberg released a poll that showed President Obama 13 points ahead of Mitt Romney among likely voters.

It goes without saying that the poll is extremely different from the other results we have been seeing lately. The FiveThirtyEight “now-cast,” which estimates the current state of the national race through a detailed evaluation of both state and national surveys, had put Mr. Obama ahead by just 1 point instead. Other methods of polling aggregation, like the RealClearPolitics average of polls, had come to about the same answer.

What to do when you encounter a poll like this?

What you don’t want to do is attribute deep meaning to it. The polls don’t move on their own; rather, they move in reaction to news as voters come to take different views of the environment around them.

There just hasn’t been very much going on in this campaign. We were all a bit spoiled by an eventful 2008 campaign cycle, and by a busy 2011 that included momentous events like the debt ceiling negotiations, the Japanese earthquake and the Arab Spring. If this campaign hasn’t been quite as slow-moving as the 1996 snoozer between President Bill Clinton and Senator Bob Dole, it has been closer to that archetype than to more spirited cycles like 1980, 1992 or 2008.

Some of the accounts that I read about the Bloomberg poll avoided that trap, describing it as an “outlier.” I don’t have any huge problem with using that term, although that still leaves you with a choice to make.

1) You could just disregard the poll entirely, or,

2) You could include it in some sort of average and then get on with your life.

My general view, as I’ve expressed before, is that you should not throw out data without a good reason. If cherry-picking the two or three data points that you like the most is a sin of the first order, disregarding the two or three data points that you like the least will lead to many of the same problems.

At the same time, it’s important to take data in context. Many polls consistently show a partisan lean toward one or another candidate. And some polls just aren’t very good, taking shortcut after shortcut that leave them far short of taking a true random sample of voters.

The FiveThirtyEight forecast model seeks to strike a balance between these ideas. If it can place a poll into context based on what it knows about the polling firm, it will use it, although the model includes a lot of checks-and-balances that are supposed to prevent the model from overreacting to any one data point.

In this case, the firm conducting the Bloomberg poll (Selzer & Company) actually has a good track record, and their previous polls this cycle had not shown especially favorable results to Mr. Obama. So the model uses the poll, just carefully.

The counterexample is when a new poll comes out from a polling firm that the model has never heard of. For instance, say there are a number of reputable polls showing Ohio roughly tied, and then a new poll comes out from a firm (hypothetically) named Boondocks Political Insight, which has never released a poll before. Their poll gives Mr. Obama a 10-point lead. In this case, the model would tend to assume that the Boondocks poll is a true outlier and that the result reflects something wrong with their methodology rather than anything informative about the state of the race in Ohio. The model wouldn’t literally throw the poll out, but it would apply a harsh “house effects” adjustment that would have the effect of diminishing its influence on the forecast.

My view is that this is a reasonable heuristic method to apply even if you aren’t operating through the formality of a statistical model. If an “outlier” poll comes from a credible polling firm, throw it in the average and don’t sweat it much; these things will happen from time to time when so much polling data is released every day. Every now and then the poll might come from a truly dubious polling firm, and then you might come closer to just ignoring it entirely.

Either of these approaches is preferable to overreacting to the poll. Some news outlets reacted with surprise to the Bloomberg poll because, they claimed, it defied the conventional wisdom that Mr. Obama had been having a bad month. However, the conventional wisdom is not always worth very much. Legitimately important events like the release of a jobs report are combined with trivialities involving Mr. Obama’s semantics. A misleadingly edited clip about remarks that Mr. Romney made at a hoagie restaurant received about as much coverage in political news outlets as the crucial parliamentary elections that Greece held over the weekend.

The problem with overreacting is that it usually begets an equal and opposite overreaction later on. If the conventional wisdom has been that Mr. Obama has been having a bad few weeks, there will be other points between now and November when the same is said of Mr. Romney, and perhaps with equally thin evidence.

If you’ve been looking at the average of polls since our forecast model was released two weeks ago, it really hasn’t changed very much. Our analysis finds that there has been about a 1-point swing in the polls toward Mr. Romney over this period. In our forecast model, this small shift in the polling has been roughly offset by the rally in the stock market (one of several economic variables the model considers), which reflects increasing investor optimism about the situation in Europe and efforts by the Federal Reserve to stimulate the economy. So Mr. Obama’s chances of winning the Electoral College have remained roughly constant, at just slightly over 60 percent, during this period.

This constancy is nothing new. There have been few major changes in Mr. Obama’s approval ratings for the past two or three years. Reaction to the shooting of Representative Gabrielle Giffords in Arizona, and to the killing of Osama bin Laden, produced temporary shifts toward Mr. Obama in the polls, while reaction to the debt ceiling debate produced a temporary unfavorable one. Otherwise, Mr. Obama’s approval ratings have remained in the mid-to-high 40s.

If news events as momentous as the Giffords or Bin Laden stories produce only relatively modest and relatively temporary changes in the polls, that doesn’t leave much room for the lead horse race story on a slow news day to matter much at all — especially when the story is literally about a horse race.

Nate Silver is the founder and editor in chief of FiveThirtyEight.

Filed under

Comments