No votes have been cast yet in the 2016 election, but there may already be one set of losers in the campaign: pollsters’ reputations. And that’s according to the pollsters themselves.
We asked people working at some of the nation’s most prominent polling outfits whether pollsters’ public image has improved or declined since the 2012 election. Of the 21 who answered, none said their public image had improved, and two-thirds said it had declined.
That was one of a few dozen questions we posed to 76 of the most prolific and prominent political pollsters.1 And we found that the people who are measuring and shaping public perception of the election — as well as Donald Trump’s Twitter account — are feeling much more positive about the work they do than they think everyone else does. (You can find the questionnaire in this PDF, all the responses on GitHub, and a list of the pollsters who responded in the footnotes.2)
Using a classic tactic in politics, many pollsters blamed the media. There were three strands to their criticism. The first is that the media make too much of bad moments for the industry, when its polls miss election results badly. (We’ve covered those moments in last year’s midterms, as well as in Israel, the U.K., Greece and Kentucky.) The second is that media organizations that aggregate polls combine the bad with the good, tarnishing all for the sins of a few. (Four pollsters said aggregators are doing badly or very badly at filtering out polls from bad polling organizations, four said they were doing OK and four said they were doing well.) And the third is that uncritical media reports of outlandish claims — such as Ben Carson’s that Egypt’s pyramids were used to store grain — leads many Americans to believe outlandish things, which pollsters are blamed for quantifying.
“Polls are wrong is a more interesting story than when the polls do well,” said Barbara Carvalho of Marist College. “Lumping all methods together distorts the accuracy of polling.”
But several respondents acknowledged that the media wouldn’t have a story if pollsters were nailing election results. “Obviously, there were several high-profile calamities in the past three years,” said Matthew Towery of Opinion Savvy. “The best I can say is this: The field is evolving, and some pollsters are succumbing to natural selection.”
Not everyone agreed that pollsters’ image is declining. Kyle Dropp of Morning Consult pointed to several positive indicators: increased scrutiny improves the work, while recruitment of data scientists and computer scientists drives innovation. He was one of three pollsters to say the U.S. polling industry is in better shape than it was after the midterm misses a year ago. Most of the 26 who responded to the question said the industry is in about the same shape as it was: Innovations in use of voter data and technology have balanced out the negative effects from declining response rates. “It’s harder to reach people but we know more about the people we’re trying to reach,” said Nick Gourevitch of Global Strategy Group.
Response rates have dipped so low over the years that eight pollsters said they were surprised polls hadn’t missed election results by even more than they have. “Some of the recent failures may be a harbinger that the response rate threat is finally materializing,” said Chris Borick of Muhlenberg College. “However, I think the relatively older electorate and marginally higher response rates in that group may continue to buffer more dramatic declines in accuracy.”
Citing reasons such as the older electorate, 14 pollsters said they weren’t surprised that polling remains reasonably predictive of election results. Several pointed to a 2012 study by Pew that found that people who don’t respond to polls aren’t different enough from those who do to skew results badly. Other factors, therefore, matter more, said Julia Clark of Ipsos: “Election accuracy is dependent on the team overseeing the work and their expertise, regardless of institution or methodology.”
Not all media critiques of polling concern accuracy. In her recent New Yorker article about polling, Jill Lepore wrote sympathetically about political scientist Lindsay Rogers’s contention that polls “are a majoritarian monstrosity.” Pollsters said they shouldn’t be blamed if the media overstate the certainty of a poll’s finding about what the majority wants, or if policy makers put undue emphasis on what polls find. “Polls are not meant to be a blueprint for policy,” said J. Ann Selzer of Selzer & Company, a Des Moines, Iowa, polling firm. “But, it is helpful to know where a majority stand — imagine a world where you did not know that.”
Here are a few other notable findings from our poll:
- Polling will get an immediate test next month with Iowa’s caucuses. We asked pollsters to name the five states that are toughest to poll during primaries. While 13 states got at least one vote, Iowa got the most, with seven. Nevada, which also has a caucus system, was next, with five. New Hampshire, which follows Iowa in the primary calendar, was third with three votes. No other state had more than one vote.
- My colleague Harry Enten noted that telephone pollsters who use live interviews consistently found lower numbers for Trump than do pollsters who use recorded voices or online panels. More pollsters (11) said they trusted the live-interview polls on this one than the interactive voice response and online polls (4). This remains very much an open question. The gap between Trump support by type of poll has narrowed. And a Morning Consult experiment to study this question suggests that Trump does worse in live-interview polls because some Trump supporters are reluctant to acknowledge their preferred candidate to another person.
- FiveThirtyEight (here and here) and Republican candidate Jeb Bush are among those who have advised ignoring national primary polls, which ask people across the U.S. who they’ll vote for, because there is no national primary, just a series of state votes, with each one affecting the next. By a slim majority (11 to 8, with one unsure), pollsters disagreed with this critique. “They are not helpful for predictions, but they tell us something about the mood of the country,” said Micheline Blum of Blum & Weprin Associates. Jay Leve of SurveyUSA disagreed. “National primary polls are worse than meaningless,” he said. “We’ve never taken one in 25 years.”
- The majority (11 to 7, with three unsure) of pollsters said voters and the media shouldn’t be paying attention to polls of hypothetical general-election matchups — Hillary Clinton vs. Donald Trump, say — at this stage of the race.
- Then there’s the big story we’ve been covering about the election: Who will win? Pollsters expect a very tight race for control of the Senate. Of the nine who predicted the number of seats Republicans will control, five said exactly 49 and four said 51 or 52. But pollsters expect the GOP to hold on to its majority in the House: Each of the nine forecasting pollsters predicted Republicans would hold at least 220 seats. In the presidential race, no one named anyone but Hillary Clinton as the Democratic nominee. Six thought Marco Rubio would be her Republican opponent, four named Ted Cruz and one named Chris Christie; no one named Trump (or Jim Gilmore).
Read more:
Harry’s Guide To 2016 Election Polls
More coverage of polling and of prior polls of pollsters