Skip to main content
Menu
Why Pollsters Think They Underestimated ‘No’ In Scotland

With two notable exceptions, opinion polls released this month about Scotland’s independence referendum vote gave an accurate picture: “No,” a vote against leaving the United Kingdom, was the steady favorite. But pollsters underestimated the extent of “no” support, making this the latest referendum with a voting-day swing toward the status quo.

“No” consistently led in polls through the end of August, usually by double-digit percentage points after subtracting the “don’t knows.” Then, at the beginning of this month, the polls tightened, and two surveys even showed a lead for “yes”: YouGov’s poll two weeks ago found a 2-point lead for “yes,” and ICM’s poll last week showed a seven-point lead without the undecideds. After that ICM poll, though, every other survey showed a lead for “no.”

When all votes were counted by Friday morning, “no” had won by 10.6 percentage points — a comprehensive win by a wider margin than any poll picked up this month.

It’s hard to say whether pollsters all erred in their calculations — perhaps with poor estimates of turnout for groups partial to one side — or whether voters changed their minds late. No media organization sponsored exit polling, which is expensive. Also, many of the Scotland pollsters use online panels, so they lacked the necessary infrastructure. “We wouldn’t have done a proper exit poll as we’re online-only so don’t have a field force to actually go and stand outside polling stations,” Anthony Wells, associate director at YouGov, said in an email Friday.

YouGov instead re-interviewed some of its panel members Thursday and reported just after polls closed that it predicted “no” would win 54 percent of the vote — 1 percentage point below the total and 2 points above YouGov’s final pre-vote poll. The company attributed this partly to shifts to “no.”

“Our recontact survey on the day suggested an on-the-day shift towards ‘no’ of about 2 points, so my prima facie assumption is that the error is down more to late swing than understating no support,” Wells said.

YouGov and Ipsos also attributed some of the change to differential turnout, and the vote numbers back that theory: Turnout was negatively correlated (R=-0.42) with “yes” support in Scotland’s 32 local councils.

Martin Boon, the director of ICM Research, said he doesn’t believe in late swings and instead attributed the underestimate of “no” support to what he called Shy Nos, people reluctant to tell pollsters they oppose independence. (Our colleague, Nate Silver, has a similar theory.)

Damian Lyons Lowe, chief executive of Survation Ltd., likened the discrepancy between polls and results to bungee jumping. Among 100 people who paid for a bungee jump, a handful are likely to get to the edge and decide they’d rather not jump. “It could have just been that,” he said in a telephone interview Friday. “We weren’t able to show that freaking-out effect.”

But maybe we all should have seen this coming. In the eight pre-Election Day surveys taken after the broadcast of “Scotland Decides: The Big, Big Debate,” “no” averaged 48.4 percent, “yes” averaged 44.1 percent and undecideds averaged 7.4 percent. Although it would be tempting to proportionally allocate undecideds to “no” and “yes” and think the race was close, history tells us that would have been a mistake.

In ballot measures, the general rule has been that voters who are undecided tend to vote with the status quo. Nate wrote about this “rule of thumb” four years ago when analyzing whether Proposition 19 would fail in California.

In Scotland, adding the undecided vote to “no” would have allocated “no” 55.8 percent of the vote. “No” ended up winning 55.3 percent.

Of course, it’s one thing to write about this in retrospect and another to predict it would happen. Several analysts did so. Last week, in trying to predict the outcome, Stephen Fisher at Elections Etc. wrote that in 12 of 16 votes on very important ballot measures internationally for which he could find data, final polls overstated support for “yes,” typically the vote for change.

Monmouth University pollster Patrick Murray, who has polled many ballot measures stateside, had noticed the trends Fisher identified, and he predicted the undecideds in Scotland would go overwhelmingly to the “no” camp. Based on this method, he tweeted a prediction that “no” would win 55 percent to 45 percent. When rounded, “no” won by that margin.

We asked several of the Scotland pollsters if they wished they had allocated the undecided disproportionately to the “no” side of the ledger because of voting history in referenda. “This is a unique event, so it would be very difficult to justify,” Lowe said. Wells, of YouGov, agreed, saying, “This referendum had already broken with precedents by showing such movement towards ‘yes’ in the final weeks, so we didn’t want to make assumptions.”

“Pollsters don’t make predictions about the future, but only give a snapshot of the story at a given point in time,” said James Crouch, a research executive at Opinium.

The pollsters, who were speaking after getting little or no sleep as the results came in, were mostly pleased with their performance, despite missing the extent of “no” support. Boon, who’d warned of a possible “polling Waterloo,” said the polls were “just about OK.” Others’ assessments were that the polls did “well” or “pretty well” or “performed relatively creditably,” or that they were “relatively accurate” — with the final polls including the actual result within their margins of error.

Gideon Skinner, head of political research director at Ipsos MORI, cited the polls’ broad agreement that there would be a “no” victory but added, “Clearly there can be improvements, and the polls will want to look at why there seemed to be a general slight underestimation of the ‘no’ lead.”

But how about those two polls that showed “yes” leading?

ICM’s Boon said the one by his firm “obviously was an outlier, and it was correctly played down.” When the ICM poll came out last week, Boon also called it an “outlier” and added, “Polls can and do go up and down, and the fuss an individual poll makes will soon be forgotten when the real result arrives.”

Survation’s Lowe, who pointed out that his firm never showed a lead for “yes,” agreed: “I strongly believe there never ever was a lead for ‘yes.’ ”

Wells wasn’t so sure that the YouGov finding of a “yes” lead was false. When the company released the poll, it accompanied the survey with dramatic rhetoric, including this line from Peter Kellner, president of YouGov: “The Yes campaign has not just invaded No territory; it has launched a blitzkrieg.”

Wells said Friday that he was confident the shift from “no” toward “yes” was genuine, pointing to tightening in most companies’ polls. “Whether it actually ever nudged over into a ‘yes’ lead — I don’t know for sure, we never will,” he said. “It may be that it did and then the banks and pension companies announcing they would pull out of Scotland moved people back into the ‘no’ column. Perhaps it was just a bit of a yessy sample that day.”

CORRECTION (Sept. 19, 1:51 p.m.): In the eight pre-Election Day surveys taken after the broadcast of “Scotland Decides: The Big, Big Debate,” “no” averaged 48.4 percent, “yes” averaged 44.1 percent and undecideds averaged 7.4 percent. A previous version of this article had the “no” and “yes” percentages reversed.

Carl Bialik was FiveThirtyEight’s lead writer for news.

Harry Enten is a senior political writer and analyst for FiveThirtyEight.

Comments