To more than one pundit, last week’s election in the United Kingdom looked like it would be the closest in a generation. But at SurveyMonkey’s Palo Alto, California, headquarters, thousands of miles away, things looked very different: Respondents to an online poll conducted by the Internet survey company from April 30 to May 6 showed the Conservatives, led by Prime Minister David Cameron, as poised for an unexpectedly comprehensive electoral triumph.1
Jon Cohen, who joined SurveyMonkey as vice president of survey research after stints at ABC News2 and The Washington Post, wrestled with whether to publish the data before the May 7 election. SurveyMonkey was a relative newcomer to election polling, having spent much of its 16 years making it easier for organizations to query their employees or customers. Cohen had intended the most recent survey to serve as an internal experiment, not be released to the public.
Still, the company had conducted an earlier poll — that it had chosen not to publish — whose findings had been close enough to other polls’ to seem like it could be right. Add the two together, and Cohen was seeing evidence that SurveyMonkey’s technology could power the kind of electoral polling that we at FiveThirtyEight write so much about.
Jon Cohen
Survey Monkey
So what to do with the latest results, which were similar to the polling consensus but different enough to be valuable to the campaign coverage?
This time, Cohen didn’t pocket the poll. Soon after sharing it with The Washington Post, Cohen learned that his poll was a whole lot closer to the actual election results than any other. It was a potential coming-of-age moment at a time when many traditional pollsters think it’s inevitable that online polls will become the industry norm. SurveyMonkey’s decision to enter the fray of a heavily polled, high-profile election created a big test for its methods, unusual even by online pollsters’ standards. In this instance, those methods worked well. But what does that mean? That its kind of online polling is ready to compete with, and beat, more traditional methods? Or that this poll was just a fluke?
For the poll, SurveyMonkey solicited responses from Britons who were taking other surveys — a random sample of 75,000 other U.K. surveys that were using SurveyMonkey’s platform. Eventually, 18,131 Britons told the company how they planned to vote.
This is very different from the methods of most online polling companies, including ones that polled about the U.K. election. Companies typically recruit panels of people who provide their demographic information and agree to be surveyed regularly. For each survey, the typical polling company selects a sample of panelists who roughly represent the electorate, at least demographically. SurveyMonkey didn’t know anything about the people it asked to participate when it asked, other than that they were in Great Britain.3
What SurveyMonkey’s respondents said looked very different from what people responding to other polls did. Every one of the roughly dozen major pollsters in the U.K. that released final pre-election numbers in the week before Thursday’s vote showed a lead of no more than 2 percentage points in the national vote share for either Labour or the Tories. Most showed a lead of 1 point or a tie — which led most forecasters, including the ones FiveThirtyEight partnered with, to predict that no party would win a majority of seats in Parliament, because of the strength of smaller parties.4 SurveyMonkey, though, found that the Tories were ahead by 6 points — a result that would have major implications, implying the Conservatives could approach and achieve a majority.5
The Conservatives won 331 seats overall, enough for a majority and 99 more than Labour. And they won the popular vote in Great Britain by 6.6 percentage points — in line with SurveyMonkey’s results.
“At root, our work regarding elections is an exercise in survey innovation,” Cohen said in an email. “One way to move past stale debates of online vs. traditional research is through elections, where at the end there are clear right or wrong answers.”
Cohen said SurveyMonkey CEO Dave Goldberg was a big supporter of the electoral experimentation. Tragically, Goldberg didn’t get to see the results. On May 1, the day after SurveyMonkey started taking responses to its poll, Goldberg died at age 47 of head trauma while exercising in Punta Mita, Mexico. Cohen worked amid his grief, attending Goldberg’s memorial service and then sending follow-up data to the Post on the same day. Cohen called Goldberg “bold.” He said, “It takes some courage to be an outlier, and his great trust in data buoyed me in this terrible week.”
SurveyMonkey’s approach has a risk: that the people who respond to its other surveys may not as a group be representative enough of the whole electorate. Just about everyone has a phone, so for polling companies that are seeking respondents by phone — until recently the predominant method — properly designing a survey to randomly dial numbers in the desired area should in theory find respondents who are representative. But what if SurveyMonkey respondents are more likely to be Tories? That could skew the results.
Two changes in the 16 years since the company was founded have eased that challenge. Just about everyone in the U.S. and U.K. is online, and fewer people are answering when telephone pollsters come calling.
Telephone pollsters who dial numbers selected randomly sometimes say they have a probability sample, meaning that respondents are selected randomly. But the lower response rates go,6 the greater the risk that random-digit dialers aren’t getting a random group of respondents, but instead are collecting information from a self-selected subgroup of people who are willing to answer a stranger’s questions. “There is no such thing as probability sampling in 2015,” Jay Leve, who runs telephone-polling company SurveyUSA, said in an email. “It’s an archaic concept.”
Many pollsters, whether because they agree with Leve or because of online polling’s price advantage (companies don’t have to pay phone bills and wages for phone-bank workers), predict that election surveys will go online. I asked U.S. pollsters last fall when they expect Internet-based political polling to overtake phone polls. The median year that respondents gave was 2020.7
SurveyMonkey’s electoral-polling methods differ in more ways than how the company finds its respondents. Cohen said the company also has found that it gets the best results when it doesn’t weight them too much. Pollsters typically adjust their numbers to account for the proportion of people from various demographic groups who respond. For instance, if a smaller percentage of respondents are black than the percentage of all voters who are black, the responses from blacks are weighted more heavily to account for the difference. SurveyMonkey does that part. What it doesn’t do is weight by likelihood that the respondent will vote — a process that has become more complex and controversial for other pollsters.
Cohen isn’t sure why SurveyMonkey’s polls do better without that kind of weighting. One possible reason he offers is that people who are enthusiastic enough to answer a survey also are likely to vote. “It’s something like, if you aren’t going to vote, you aren’t going to answer the survey,” he said.
SurveyMonkey has polled elections before. It correctly identified the winner in 48 of 50 states and Washington, D.C., in the 2012 U.S. presidential election, both gubernatorial races in 2013, all 36 Senate races in 2014, 33 of 36 gubernatorial races that year, plus the Washington, D.C., mayoral race.8
However, Cohen hadn’t published any of the company’s electoral survey results before polls closed since joining SurveyMonkey. In 2014, he shared the company’s numbers with partner NBC News before voting returns came in, but the SurveyMonkey findings weren’t released until after the election. “I didn’t know enough about 73 races to know we were right, so I didn’t want to publish, but I felt extraordinary confidence in our methods,” Cohen said.
Fourteen times in our phone conversation on Friday, Cohen used the words “confident” or “confidence” — words you won’t see much in analysts’, pollsters’, industry’s and forecasters’ befuddled attempts to make sense of their big electoral miss in Great Britain. Cohen was reluctant to speculate too much about what went wrong in an election in another country by pollsters he doesn’t know, but he did suggest that pollsters might have been herding — intentionally choosing which results to release, and how, so as not to veer too far from the consensus. One pollster, Damian Lyons Lowe of Survation, admitted to having “chickened out” of releasing a late poll showing a 6-point Conservative lead — something, he wrote, that he’s sure he’ll “always regret.”
People who were skeptical of SurveyMonkey’s methods before last week, though, see not 18,000 surveys producing the right result, but one survey getting the right result, perhaps by luck. In other words, they see a sample size of one and lots of uncertainty about whether SurveyMonkey can pull it off again.
“Over time, given a SurveyMonkey track record, observers may be in a position to conclude that SurveyMonkey has a secret sauce that makes them the envy of the polling world or that they just got lucky in Britain,” Leve said. “But we won’t know that for some time.”
After all, there are only so many elections in which pollsters can test their methods — although SurveyMonkey may be able to poll more elections than most companies, since it has survey takers in more than 190 countries, Cohen said.
Other skeptics say there’s already evidence that SurveyMonkey got lucky this time. They point out that although the SurveyMonkey poll was right about the size of the Conservatives’ margin of victory over Labour in British vote share, it was off on the percentage of the Great Britain vote that the party earned. The poll said the Conservatives would win 34 percent; they ended up getting 38 percent. If you average the SurveyMonkey poll’s error on each of the biggest six parties’ vote share by that measure, it performed in line with other pollsters. “This isn’t particularly close to the result,” said John Curtice, president of the British Polling Council.9
“They might have got the lead close to right, but the levels for nearly all the parties including Conservatives and Labour are terrible,” Stephen Fisher, a political sociologist at the University of Oxford who forecast the election, said in an email.
Cohen said he welcomed peer evaluations and the discussions they’d engender, starting with those that will take place at this year’s conference of the American Association for Public Opinion Research, the U.S. pollster group, that begins this week. “Judging surveys by their results is a positive sign of where we are as an industry — it shows that we’ve moved past dismissiveness,” he said in an email. He pointed out that the poll got consistent results: Conservatives led in six of the seven days it was in the field. “I don’t see ‘luck’ — or ‘chance’ — as the simplest explanation here.”
Cohen doesn’t claim to have solved all the polling industry’s problems. “I think political polling is ripe for transformation, and we have a lot of the attributes in SurveyMonkey that give us hope we can be part of that transformation,” he said. “I don’t sit here today and say we’ve figured out the world.”