For our second poll of leading U.S. political pollsters, we asked about ethics. The pollsters who answered, even those who asked for anonymity, said they follow basic ethical principles such as not copying others’ work or letting campaigns dictate results. But many held doubts about their peers’ ethics, and about the media’s ability to parse honest, good polls from dishonest, lousy ones.
As with our first poll, we contacted the most active political pollsters in our database.1 This time, 24 responded, on questions about ethics and about FiveThirtyEight’s pollster ratings and Senate forecast. Their answers reveal the thinking of two dozen people who conduct many of the polls that help inform horse-race coverage of major elections. Their work is a primary factor driving our and others’ models forecasting November’s vote for Senate control. (You can see the poll and download the full results of both polls on GitHub.)
Earlier this year, my colleague Harry Enten reported on evidence he found suggesting that some pollsters were copying others’ results, revising their numbers to fall into line with those of their peers. I asked our pollsters if any ever do that. All 22 who answered said they don’t. “We are paid to provide our findings and not to be influenced by others,” Bernie Porn of EPIC-MRA said. Another pollster said, “We may be wrong, but we’ll live with that rather than being dishonest.”
John Anzalone, of Anzalone Liszt Grove Research, drew a distinction between revising numbers to match others’ and rejecting polling results that don’t pass the smell test. He pointed out that even an optimal poll will get poor results — off by more than the margin of error — one out of 20 times. “That is just statistics,” Anzalone said. When his firm suspects that’s happened, sometimes “we will go back into the field at our own cost to check.” Conducting a brand-new poll is not at all the same as changing a poll’s findings.
One respondent wasn’t convinced we’d get meaningful results from the question, even though we granted anonymity. “Um, you think anyone is going to answer this honestly?” said the pollster, who didn’t want to be named.
When we asked the same question about other pollsters, our respondents were more inclined to suspect copying. Of the 14 who answered, eight thought some pollsters adjusted their numbers to match published polls’ results. Two people cited Enten’s article as the reason for their belief. Another said, “I’ve just noticed over the years that certain organizations have wild fluctuations in their numbers that often follow the release of a poll with numbers that run counter to what they had recently published.”
Julia Clark of Ipsos answered “No” to the question, saying adjusting results that way would “fly in the face of any kind of good practice, and would become obvious very quickly (and discredited).” However, she added that paying some attention to others’ work is a different matter. “With the renewed focus on Bayesian statistics” — Clark cited this recent New York Times article as an example — “it is inevitable that pollsters will begin to take cues from the marketplace. The form this takes will vary enormously by pollster.”
Many pollsters agreed with Clark that consulting their peers’ results was sensible. Of 21 who answered a question about whether they do this, four said “Yes” and six said “Sometimes.”
“We do it to anticipate potential questions or backlash, and to help our clients plan for this,” Clark said. “It would be crazy not to look at where the market is before publication!”
Speaking for the narrow majority who don’t check others’ work, Andrew Smith of the University of New Hampshire said, “Trust your methods, trust your numbers.” Another pollster added, “We just take our best shot. I don’t really want to care what other polls say.”
We also asked our respondents who poll for campaigns whether they ever let those campaigns dictate weighting or other factors that can affect results. All 15 who answered said they don’t. “My job with any client is to tell them what they need to know and not what they want to hear,” Brad Coker of Mason-Dixon Polling & Research, Inc., said.
Several pollsters described how their relationships with campaigns can get a little complicated. Sometimes campaigns want to try out different assumptions, such as about voter turnout, to see how they affect the poll’s results. Pollsters said they are OK with that, so long as these alternative scenarios are for internal consumption only. But if the campaigns want to publish results, the pollsters say they insist the results come from the model they believe. And if the campaigns push back? J. Ann Selzer, of Selzer & Company, considered that hypothetical: “If a campaign tried to publish a poll I did not think reflected an accurate picture, yet they wanted to portray it that way, I reserve the right to comment and clarify. Publicly. Or take my name off the poll, I suppose.”
But again, several pollsters said they think others in the industry don’t adhere to those standards. I asked if there are any polls we shouldn’t be including in our model. “I don’t believe party-affiliated polling firms should be included in the models,” Berwood Yost of Franklin & Marshall College said. “I have a concern that partisans are increasingly producing polls in the hope that they can affect the poll averages in a race.”2
Another pollster said trying to forecast elections simply by averaging polls would fail because there are “too many mysterious polls being released. I think some are being financed by partisan groups without disclosure. A straight average would be skewed simply by one side putting out more polls than the other.”
It’s not just election forecasts that could be unduly swayed by partisans, pollsters said. They also fear the media is too credulous about reporting polls. “Some polls are so biased that they shouldn’t be reported,” Yost said. Anzalone said, simply, “Stop publishing shitty polls.”3
More than partisan bias can undermine a poll’s quality. Both Porn and Darrel Rowland of the Columbus Dispatch said reporters should try to see polls’ question wording and order, because each can influence how people respond. If pollsters won’t share that information, reporters shouldn’t publish their work.
Sometimes, reporters publish results of polls — then hear from polling organizations that they have recalled the poll, like a faulty car. All but three of the 20 pollsters who responded to our question about it said recalling polls was sometimes appropriate. “There is so much emphasis on speed these days that errors inevitably creep in,” Clark said. “It is important to acknowledge them. If that takes the form of a ‘recall,’ so be it.” However, she didn’t think a recall was practical — once a poll’s in the public domain, it can’t be unpublished.
Not everyone focused on the bad actors and errors in their midst. Many emphasized the good, innovative work being done. Pollsters provided lots of nominations when asked to name a polling organization other than their own that is doing the best work. Four named Pew Research Center, with one specifically citing its nonelection work. “Pew has become the gold standard of polling,” Anzalone said.
Selzer and Gallup each got two votes. Public Policy Polling, CBS News/New York Times, NBC News/Wall Street Journal, Ciruli Associates and Democratic pollsters Stanley Greenberg and Michael Bocian each got one.
Ciruli got its vote from a peer who’d never heard of the company but saw it received an A+ in our ratings. That was typical of the generally positive reaction the pollsters had about our ratings — with some reservations.
Of 22 pollsters who answered our question about whether our rating of their organization was fair, just six said it wasn’t. Eight said it was, and six chose “Other.” It’s possible that pollsters who like our ratings are more likely to participate in our survey and to answer this particular question than are those who don’t like them.
Some pollsters raised concerns about the ratings, including:
- Those who poll for campaigns or otherwise conduct private polls don’t get credit for those.
- Pollsters who don’t allocate undecided voters to candidates may make different predictions about the final vote margin than their official polling numbers suggest.
- Our database goes back to 1998, so pollsters that have changed their methods recently and gotten improved results don’t get full credit. (We do weight more recent polls more heavily.)
- Pollsters “are so close together that the grading cuts are arbitrary,” Patrick Murray of Monmouth University said.4
Selzer said she agreed with our rating of her company. “You did not contact us for dates, data — anything. You found what was published. No one would monkey with the findings — they were completely data-driven.”
Then again, Selzer’s grade is one reason she might like our ratings: Her company gets an A+. Nearly all of the pollsters who agreed with our rating of their polling organizations had grades of A- or better; nearly everyone who disagreed scored a C or lower.5
Anything you want to ask our pollsters? We’ll be sending them another survey soon, and are open to suggestions. (We also asked the pollsters to suggest questions for one another, and will use some of those.) Email suggestions to cbialik@fivethirtyeight.com or leave them in the comments.