Democrats’ odds of holding the Senate are declining based on the latest polls. But the race is close. Republicans are counting on narrow leads of just a percentage point or two holding up in some of the most important races, like those in Iowa, Colorado, Georgia and Alaska.
Last month, I examined the possibility that Senate polls could prove to be biased or “skewed” against Democrats. If the polls have even a modest bias against Democrats, the party’s chances of keeping the Senate would be more like 50-50. And if the polls have a more severe anti-Democratic bias, it would be favored to keep its majority.
There is, indeed, some precedent for this, and we don’t have to go back far. In 2012, the average Senate poll underestimated the Democratic candidate’s actual performance by 3.4 percentage points.
However, there have been just as many years, like 1994 and 2002, when the polls proved to have a pro-Democratic bias instead. And over the long run, there’s been no long-term bias toward either Democrats or Republicans in Senate polls. Many efforts to predict the direction of bias in the polls, from the silly to the sophisticated-seeming, have failed in the past. The direction has often reversed itself from one cycle to the next.
Some Democrats, however, are not asserting the polls will have an overall bias: Instead they’re focused on particular states, especially Colorado, which might just be enough to let them keep the Senate. In 2010, Democratic candidate Michael Bennet won the Senate race there despite narrowly trailing Republican Ken Buck in most surveys. And President Obama won Colorado by a slightly wider margin than the polls predicted in 2008 and 2012. If Democrats win Colorado, their chances of keeping the Senate would rise to 40 percent from 27 percent, according to the FiveThirtyEight model’s simulations.
So, let’s ask a few questions:
- What has been the historical bias of the polls in states other than Colorado?
- Has the direction of this bias been consistent from election to election?
- Are there other factors that predict state polling bias?
The data I’ll be using is from FiveThirtyEight’s pollster ratings database, which covers polls conducted in the final three weeks of campaigns since 1998. (You can download the data here.) I’ll be using polls from all presidential, Senate and gubernatorial races in which a Democrat and a Republican were the top two finishers.1 I’ll also use polls of the U.S. House, but only in states like Alaska and Delaware that have just one at-large congressional district. Bias is calculated as the difference between the actual election result and the polled margin. For example, if the Democrat is projected to win by 5 percentage points in a poll and she wins by just 2 points, that poll would have a 3-point Democratic bias.
Here are the numbers:
There’s a lot going on in this chart, which my colleague Allison McCann put together. The states are sorted by the average bias since 1998, with states where polls have been biased toward Democrats on top and those where they were biased against Democrats on the bottom.
Democrats, indeed, have outperformed their polls in Colorado — by an average of 1.6 percentage points since 1998. That closely matches Republican Cory Gardner’s lead over Democrat Mark Udall in the polls this year. So, if they have the same bias again, the race will be a tossup. The polls have also been biased against Democrats in Iowa, another key Senate race, by an average of 1.5 percentage points.
But that’s where the good news ends for Democrats. In the other eight states with the most competitive Senate races, the polls have historically had a bias toward Democrats. This has been by a trivial amount (0.2 percentage points) in North Carolina and New Hampshire. But it’s been more significant elsewhere: by 1 or 2 percentage points in Arkansas, Georgia, Kansas, Kentucky and Louisiana — and by a whopping 7 points in Alaska. If I adjust our forecast according to the historical bias in each state — for instance, adding 2 points to Udall’s projection in Colorado but subtracting 7 from the Democrat Mark Begich’s in Alaska, Democrats’ odds of keeping in the Senate would decline slightly overall, to 24 percent from 27 percent.
This testifies to one of the problems with the arguments over “skewed” polls. Like other polling-related debates, they often rely on cherry-picked evidence. If Democrats want to go down the path of evaluating the historic bias in state polls, they’ll wind up in a slightly worse place at the end; the gains they make in Colorado and Iowa are offset by further deterioration in Alaska and to a lesser extent in Georgia and Louisiana — some of which they’ll have to win to keep the Senate.
But this is premature. Is there any consistency in state poll bias in the first place? For instance, does the historical bias in Colorado and Alaska tell us anything about what it might be this year?
There is some consistency from cycle to cycle. About half the polls in our pollster ratings database were conducted between 1998 and 2007; the other half have been in elections since 2008. So we can cleave the data into halves and see whether the bias ran in the same direction in each period. In 35 of the 50 states, it did.
However, there’s also quite a bit of noise — the correlation is a modest .32.2 Furthermore, some of it is caused by the way I’ve handled undecided voters in this calculation. If you treat them differently, the consistency in state poll bias drops to almost nothing — see the footnotes for more about that.
As some of you may have noticed, however, there’s a much stronger relationship between poll bias and state partisanship. The five states where Republicans have most outperformed their polls are Wyoming, Alaska, Idaho, Oklahoma and North Dakota. Democrats have done best relative to their polls in Hawaii, Delaware, New York, California and Nevada. Republicans have tended to beat their polls in red states while Democrats have beat theirs in blue states.
In the chart below, I’ve compared the direction of state poll bias in elections since 2008 against the results of the 2004 presidential election between George W. Bush and John Kerry. (The Bush-Kerry numbers are taken relative to the national popular vote margin. Kerry did not win Ohio in 2004 — however, he performed slightly better there than he did nationally.) The reason I use 2004’s results rather than some later year is to test whether partisanship had predictive power out of sample — in elections held from 2008 onward, at least four years after the Bush-Kerry results were known.
This correlation is quite strong: it’s .66.3 Furthermore, once you account for it, there’s no longer much meaningful information conveyed by the past direction of polling bias in the state; partisanship rules instead. If a red state had a Republican bias in the past — as Louisiana did from 1998 to 2007 — you’d nevertheless predict it to have a Democratic bias in future years.
This is potentially bad news for Democrats: Think about where the key Senate races are being held this year. They’re all in purple or red states. No Democrat from a solidly blue state is at risk of losing his seat, and Susan Collins, the Republican senator from blueish Maine, is not going to lose hers.
If past patterns hold, you’d expect polls to be unbiased in purple states like Iowa and Colorado — and biased against Republicans in those like Kansas and Alaska, two hard-to-poll races that are instrumental to Democratic hopes of keeping their majority.
Some of this may have to do, as Sean Trende describes, with the behavior of undecided voters. In Kentucky, for example, Democrat Alison Lundergan Grimes’s share of the vote has held fairly steady at 42 or 43 percent — however, Republican Mitch McConnell’s has increased from 42 or 43 percent to about 48 percent. A good candidate (like Grimes) in a tough environment (like Kentucky for a Democrat in a federal race) may capture her voters early on but then run out of voters to persuade. (As I’ve mentioned, the handling of undecided voters also matters somewhat in the bias calculation — see the footnotes for more about that.)
Also when using this version of the bias calculation, past polling bias is no longer a statistically significant predictor of future polling bias in a state:
More broadly, this may be an argument for looking at state “fundamentals” along with the polls — as the FiveThirtyEight model does (although the fundamentals receive very little weight in the model at this stage of the election). Among the most important fundamentals variables are simply past election results. For incumbent senators, what was their margin of victory in their previous race? And has the state voted Democratic or Republican in past presidential and congressional elections?
The Senate fundamentals do not favor Democrats this year. The elections are being held in a red-leaning group of states. Several of the Democratic incumbents won fairly narrowly in 2008, when they were last on the ballot, despite it being a terrific year for Democrats nationally. Republicans will win the Senate if they flip the Democratic seats in Montana, South Dakota, West Virginia, Alaska, Arkansas and Louisiana, while holding all of their own.
While there are some legitimate reasons to worry that the polls could be biased against Democrats this year, this is one reason to think they could be biased toward Democrats. Indeed, while the polls have moved toward the red-leaning fundamentals in several states like Kentucky, there’s an argument the GOP is still underperforming them nationally. The generic congressional ballot currently favors Republicans by about 2 percentage points (although it’s been wildly inconsistent from survey to survey), less than the 5-point advantage that the opposition party has had on average in midterm years. This is despite the dour mood of the country and Obama’s unpopularity.
Although you should trust the polls at this point, they suggest a range of outcomes are possible. It’s plausible the polls showing a larger Republican advantage on the generic ballot (like the ABC News/Washington Post poll, which puts the GOP up 6 points) will prove correct. If so — if we wind up with a strongly Republican-leaning national environment — we’ll know fairly early Tuesday night that Republicans are going to win the Senate. They’ll probably win some states (North Carolina and New Hampshire on the Senate side, along with a number of gubernatorial races) where they slightly trail in the polls. Gardner could win by high single digits in Colorado, as could Joni Ernst in Iowa.
The FiveThirtyEight model accounts for the possibility that the polls could be biased in either direction. That they could be biased against Democrats is part of what’s keeping Democrats’ hopes alive. But they could be biased against Republicans, too. Historically, that’s been the case more often than not in red states like the ones where some of the most crucial Senate races are being held.
CORRECTION (Nov.3, 10:57 a.m.): A previous version of this article said that since 1998, polls have been biased against Democrats in Colorado by an average of 2.2 percentage points and in Iowa by 1.8 points. They have been biased by 1.6 points in Colorado and 1.5 points in Iowa.
