What To Make Of Kentucky’s Polling Failure

It feels like déjà vu all over again: The polls in a major election were off by a wide margin. The surveys leading up to Tuesday’s gubernatorial election in Kentucky pointed to a close fight between Democrat Jack Conway and Republican Matt Bevin, with Conway holding a slight edge in all but one poll. Instead, Bevin won 53 percent to 44 percent.

It was only the latest in a series of high-profile polling errors over the past couple of years.

So is polling broken and not to be trusted heading into the 2016 presidential election, as some have argued? Not quite. A more nuanced look at the results from Kentucky and Mississippi, which re-elected its governor on Tuesday, suggests that surveys can still be informative, but should never be taken as gospel.

Just as in 2014, the polling underestimated Republican candidates in Kentucky in 2015, but did so in a fairly uniform fashion. On average, SurveyUSA and Western Kentucky University (WKU) missed the final margin by 13.4 and 11.1 percentage points, respectively, in the elections for agricultural commissioner, attorney general, auditor, governor, secretary of state and treasurer.

A look at the table above reveals that the polls and the results for the different races were highly correlated (0.96 for WKU and 0.97 for SurveyUSA). It’s as if the electorate became about a dozen percentage points more Republican between when the polls were taken and Election Day.

It’s not yet clear whether pollsters simply projected that more Democratic voters would show up than actually did or whether undecided voters broke overwhelmingly for the Republican candidates. The former suggests an electorate modeling problem that could be a big problem during the presidential primaries, when turnout is low. On the other hand, trouble modeling the electorate would be less of an issue in the 2016 general election, when turnout is at its highest.

However, if undecided voters broke toward the Republicans and the “fundamentals” — Kentucky is a very Republican-leaning state in federal elections — that could be a sign that a candidate hitting 50 percent in general election polls is a bigger deal than we previously thought. Remember, polls showed Democrat Mark Warner winning the 2014 Virginia Senate race easily with 50 percent of the vote; Warner earned almost exactly 50 percent, but he barely won.

Yet, I would be careful of making too much of the Kentucky results. Only three polls not sponsored by a candidate came out during the final three weeks of the campaign. That’s far less polling than was conducted in other recent polling mishaps, such as in Israel and the United Kingdom over the past year. The Kentucky results match most of the bigger misses in the U.S. during the 2014 midterm elections, such as in the Maryland gubernatorial race and Virginia Senate election, when few polls were released during the final weeks of the campaign.¹ That’s a good thing for 2016, when the most highly anticipated races will have lots of polls in the field.

Moreover, the one poll that came out in Mississippi before Tuesday’s elections didn’t underestimate the Republican candidates.

Mason-Dixon’s average error in the races for attorney general, governor, lieutenant governor and secretary of state was 5.0 percentage points,² and the largest error (in the secretary of state’s race) overestimated the Republican’s margin. Of course, it’s also just one poll.

It would be silly to claim that everything in the land of polls is great. Errors like Kentucky shouldn’t happen very often, and they seem to be occurring more frequently these days. But for those of you who care about the accuracy of polls, I would counsel you not to freak out.