Skip to main content
The Polls Might Be Skewed Against Democrats — Or Republicans

This year’s Senate election is close. Republicans have the clearer path to a majority, but it’s a treacherous one. Republicans are favored in Iowa and Colorado, for example, but not by much. If Democrats eked out victories in those states and picked up the seat in Kansas (by persuading independent Greg Orman to caucus with them), they could control 50 Senate seats and keep their majority. Democrats could also still pick up seats from the GOP in Georgia and Kentucky. And Republican gains in Arkansas and Louisiana are probable but far from certain.

So, we’ve not quite reached the point — as we did in the presidential race late in 2012 — when there would have to be a systematic error in polls across the country for the underdogs to prevail. Relatively modest errors in the right states would do the trick for Democrats. These errors occur more often in Senate races than in races for the presidency.

More Politics

Nonetheless, we’ve reached a stage in campaign season when Democrats have begun to complain that the polls are biased against them. There’s a long tradition of this sort of “unskewing.” The trailing party will say that its internal polls tell a different story or that its turnout operation will save it. It will critique each poll’s demographic cross-tabs. (Because most polls report breakouts for a dozen or more demographic groups, all with small sample sizes, there’s almost always something to argue about.) The party will point toward previous instances when it outperformed its polls. As a last resort, it’ll claim that this election will be different somehow.

Usually this doesn’t end well for the unskewers. In 2004, some Democrats asserted that John Kerry would outperform his polls because undecided voters would break toward him. Instead, George W. Bush won by a slightly wider margin than the polls predicted. Throughout 2012, conservatives argued that the polls had a Democratic bias. The polls did have a bias — but it was a Republican one.

Democrats may not be wrong. The polls could very well be biased against their candidates. The problem is that the polls are just about as likely to be biased against Republicans, in which case the GOP could win more seats than expected.

silver-datalab-pollskew-1In a number of elections, including 2012’s, Senate polls had a systematic bias toward one party. But the direction of the bias has been inconsistent, favoring Democrats in some years and Republicans in others. The chart here depicts the average partisan bias in Senate polls of likely voters conducted in the final three weeks of campaigns since 1990. (For raw data from 1998 onward, see here; for 1990 through 1996, see here). A year indicated as having a Republican bias means the GOP underperformed its polls. A year shown as having a Democratic bias means the Democrats underperformed theirs instead.

In 2012, Senate polls had a Republican bias of about 3.5 percentage points. That means in a state where the polling average showed the Republican ahead by a point, the Democrat would be expected to prevail by 2.5 points instead. If there’s the same bias in the polls this year, Democrats would be very likely to keep the Senate.

But as I mentioned, this bias has flipped back and forth. There was also a Republican bias in 1998 and 2006. But there was a Democratic bias in 1994 and 2002. On average since 1990, the average bias has been just 0.4 percentage points (in the direction of Republicans), and the median bias has been exactly zero.

Democrats might argue that a Republican bias has been evident in recent years — even if it hasn’t been there over the longer term. But the trend is nowhere close to statistically significant. Nor in the past has the direction of the bias in the previous election cycle been a good way to predict what it will look like in the next cycle. For several consecutive midterms, the bias ping-ponged between the parties: There was a big Democratic bias in 1994, then a big GOP bias in 1998, then a Democratic bias again in 2002, then a Republican one in 2006.

There are a number of other examples, apart from Senate races, of polling bias being unpredictable:

  • Conventional wisdom once held that undecided voters would break toward the challenger (hence some of the predictions for a Kerry victory in 2004). But there has been no evidence for the so-called incumbent rule for almost a decade. It may be that to begin with, the theory relied on a poor interpretation of the data. Or it may be that the pollsters found ways to work around the rule, such as by reducing the number of undecideds by “pushing” voters to reveal their candidate preferences.
  • It was once thought that polls overrated the performance of African-American candidates — the Bradley Effect. But this, too, has disappeared. Barack Obama, if anything, has historically done slightly better than this polls.
  • The generic congressional ballot was previously thought to have a Democratic bias. But it didn’t have one in 2008, 2010 or 2012. Each year featured a modest Republican bias instead.

One reason for the lack of a long-term bias is that pollsters are capable of making adjustments to their surveys. What adaptations have they made this year? In contrast to past elections, the vast majority of polls in 2014 have either included cellphones or been conducted wholly or partly over the Internet. The failure to reach voters who can’t be reached by landlines was probably introducing Republican bias into the polls before, but that may be corrected now.

What about the “ground game”? There’s credible reporting that Democrats are investing more into it this year, and that could help them. But the pollsters, at least as a group, are not expecting the sort of turnout gap they did in 2010. That year, the average poll had Republicans doing about 6 percentage points better among likely voters than among registered voters — a historically large difference. The average poll we’ve tracked this year has shown about a 3-point gap (favoring Republicans) instead — in line with the historical average in midterm years.

And remember, the question is not which party has the stronger ground game, but whether a stronger ground game will lead to benefits that aren’t reflected in the polls. An analogy can be made to stock prices. You might think that Apple makes cool, market-leading products, but that isn’t a reason to buy its stock — pretty much everyone else agrees, and that’s reflected in its stock price. It’s much more challenging to prove that the market is undervaluing (or overvaluing) the stock.

One problem with attempting to compensate for bias is that you can potentially overcompensate for it — like a golfer who alternatively shanks shots to the left and right of the fairway. Republican voters report having higher levels of enthusiasm this year (although it’s down for both parties from 2010), and Democrats have grown more reliant upon voting groups that sometimes decline to participate at the midterms. If you introduced a new turnout model this year because your polls had a Republican bias in 2012, you could wind up with a Democratic bias this year if those marginal voters only show up in presidential years.

Nor are the polls asserting we’re in the sort of substantially Republican-leaning national environment that prevailed in 2010, a year when Democrats lost Senate races in purple states such as Ohio and New Hampshire by double-digit margins. Instead, the generic congressional ballot shows a rough tie between Democrats and Republicans on a likely voter basis (or perhaps puts the GOP ahead by a percentage point or so). This is much different than in 2010, when it gave Republicans a 7- or 8-point edge.

The problem for Democrats is that many key Senate races are being contested in red states. They could lose the Senate by losing seats in Alaska, Arkansas, Louisiana, Montana, South Dakota and West Virginia — states that voted for Mitt Romney by an average of 19 percentage points in 2012. In purple states such as New Hampshire, Iowa, Colorado and North Carolina, Democratic candidates are holding their own, slightly trailing in some but leading in others. But Republicans would not need any of those if they swept the races in the profoundly red states (including in Kansas and Kentucky, where their candidates are vulnerable based on the polls).

And yet for all that, the GOP edge is slim. The FiveThirtyEight model accounts for two types of error in its forecasts: state-by-state error and national error. National error accounts for the prospect that the polls could be systematically biased toward one or the other party on average in all states; state-by-state error, as the name implies, pertains to idiosyncratic error in one state only. Ordinarily, when we run the model each day, both factors are treated as random variables and both contribute to the uncertainty in the forecast. But for this article, I’ve taken the national bias as a known quantity instead. I’ve also assumed the election will be held today instead of in three weeks, which further reduces the uncertainty.


Suppose the polls have just a 1 percentage-point Republican bias, for example, in the average Senate race. In that case, Iowa and Colorado would go from slightly favoring Republicans to being more like true tossups. Democrats would also be slightly more likely to pull off an upset in a state such as Georgia, where the GOP has only a tenuous advantage. Overall, the probabilities for Senate control would come out to almost exactly 50-50.

However, the Democrats’ path would become much steeper if there’s even a modest Democratic bias in the polls. As the race stands, they’ll need to win a couple of states where they are slight underdogs. If they become somewhat heavier underdogs in those states, their chances of getting “lucky” in two or more states on election night will diminish toward zero. According to our model, Republicans would be 92 percent favorites to take the Senate if the polls have a 1-point Democratic bias, and all but certain to do so if it’s any larger than that.

What if the polls prove to have no bias? Our model shows Republicans as about 75 percent likely to win a Senate majority. This may seem confusing: Doesn’t the official version of FiveThirtyEight’s model have Republicans as about 60 percent favorites instead? Yes, but some of the 40 percent chance it gives Democrats reflects the possibility that the polls will have a Republican bias. If the polls were guaranteed to be unbiased, that would would make Republicans more certain of winning. (An additional portion of the Democrats’ 40 percent chance reflects the possibility that the polls will drift toward Democrats between now and Election Day.)

Put another way, the FiveThirtyEight model already accounts for the possibility that the the polls could be biased against Democrats. It also accounts for the prospect that they could be biased toward them. In that case, Republicans could win New Hampshire or North Carolina, where they currently trail in the polls. In about 16 percent of our simulations, Republicans wound up with 54 or more Senate seats. That would probably qualify as a “wave” election.

It’s reasonable — indeed healthy — to be skeptical about the polls. Many of the states on the ballot this year present unique polling challenges. Many have a large number of undecided voters. And the quality of the polling is mixed. Historically, the error in polls is considerably larger than their margins of error alone imply.

Just don’t assume your favorite candidate will catch all the breaks. Sometimes the polls are biased — in the opposite direction of what you were hoping for.

Nate Silver is the founder and editor in chief of FiveThirtyEight.

Filed under , , , , , , , ,

Comments Add Comment