About a year ago, I was on a book tour in Edinburgh and was asked by a couple of reporters about Scotland’s upcoming vote on whether to secede from the United Kingdom. The “yes” vote (a vote to secede) had “virtually no chance” of prevailing, I said. “For the most part it looks like it’s a question of how much the ‘no’ side will win by, not what the outcome might be.”
Now that the “no” side has won by what looks to be a definitive margin, I suppose I should be touting that prediction. But despite the outcome, it was one of the worst predictions I’ve made. I’d spent all of 15 minutes studying the issue before weighing in. That’s not enough — and it’s not what we’re all about at FiveThirtyEight. We take predictions seriously, and there’s usually a heck of a lot of research involved before we make one. My Scotland prediction failed that test.
So I don’t pretend to have any particular authority to discuss Scotland’s results. But the “no” side’s margin of victory, by 10 to 11 percentage points, might give us some pause.
To be clear, most of the polls had “no” ahead in the closing days of the campaign. But they suggested a considerably closer election than actually happened. There’s a parallel to the final days of the 2012 U.S. presidential election, when the polls showed Barack Obama ahead of Mitt Romney. Those polls “called” most outcomes correctly but nevertheless had a significant bias, underrating how well Obama — and Democratic candidates for the Senate — might do.
Out of all of the polls in our (soon to be publicly released) polling database (which includes polls for the Senate, the presidency, and gubernatorial and U.S. House campaigns), the average survey understated the Democratic candidate’s performance in 2012 by almost 3 percentage points. Had the error run in the opposite direction — had Republicans outperformed their polls by that margin instead of Democrats — Romney might have won states like Colorado, Ohio and Florida and possibly have become president. The GOP perhaps would not have taken the Senate, but Republicans would have been highly competitive in states including Montana, North Dakota, New Mexico, Massachusetts, Wisconsin, Ohio and Virginia and made it very close.
Suspend your disbelief for a moment, and imagine I’ve persuaded you that the margin between actual and predicted results — not the number of correct “calls” — is what counts in polling. What then accounted for the mediocre results in Scotland?
I have only a theory: Scotland’s results may have had something to do with the “Shy Tory Factor.” This was the tendency of conservatives (Tories) to outperform their polls during a number of U.K. elections in the 1990s and especially in the U.K. general election of 1992. The idea is that conservatives were less enthusiastic than Labour voters and therefore less likely to declare their support for a conservative government to pollsters. Nevertheless, they turned out to vote.
U.K. pollsters responded to these elections in a variety of ways, including by weighting their results based on voters’ party preference in prior elections. But this may have been a patch that failed to address the underlying issue: Voters are not equally likely to respond to polls; those who are more enthusiastic about an upcoming election are more likely to do so.
This potentially leads to a double-counting of enthusiastic voters if turnout models are not applied carefully.
The problem could become worse as response rates to polls decline. Furthermore, many polls of the Scottish independence referendum were Internet-based, and some of those polls did not use probability sampling, historically the bedrock for demographic weighting. A YouGov poll earlier this month, for example — one of the few to show the “yes” side ahead — did so only because of its weighting procedures. On an unweighted basis, it had “no” ahead by about 6 percentage points.
Almost everyone in Scotland voted in the referendum. Less enthusiastic voters — Shy Unionists? — may have been missed by pollsters, but they may have made the difference.