Trump Supporters Probably Aren’t Lying To Pollsters

Could the polls be underestimating Donald Trump’s support? That’s the conclusion of what I’ll call the “shy Trump-ers” theory, which holds that public opinion surveys undersell Trump because some of his supporters are unwilling to admit they’re backing The Donald. The supposed evidence for this comes from comparing different types of polls: Through much of this Republican presidential primary, Trump did worse in polls that used live, human interviewers than in polls that don’t. The same gap appears in general election polls testing Trump against Hillary Clinton.

But when you compare the polls to votes — Republican primary and caucus results — the bulk of evidence suggests that there aren’t many shy Trump supporters.

First, let’s look at how the polls did collectively (before we separate out the live-interview surveys). We have polling and voting results from 34 caucuses and primaries. For each race, I looked at Trump’s support in each poll in the final three weeks before the contest and then allocated undecided voters.¹ In the table below, negative values indicate that Trump underperformed his average poll in a state, and positive values indicate that he outperformed his average poll. Based on previous races, we generally expect candidates to meet their average poll support once undecided voters are divvied up.

Trump did worse than the polling forecast in 19 states; he did better in 15 states. That hardly suggests that Trump outperforms his polling. Still, the difference isn’t so great that we can say Trump usually underperforms his polling. It’s a fairly even split, with Trump missing his average poll by just 1 percentage point in the median state. Two of Trump’s worst performances relative to the polls were in Kansas and Iowa — both states held low-turnout caucuses, which Trump won’t have to deal with in the general election. Overall, Trump’s percentage of the vote versus the polls is about what you’d expect of the average politician.

Trump was far more likely to outperform his polls in the final stretch of primaries before he became the presumptive nominee — New York through Indiana. Through the Wisconsin primary on April 5, Trump underperformed his polling average by a little less than 3 percentage points in the median state. After Wisconsin, Trump outperformed his average poll by nearly 7 points. This trend culminated May 3 in Indiana, where Trump did nearly 10 points better than the average poll conducted in the final three weeks before the primary.

These later contests could be signs of a shy Trump-er effect. Trump went from winning pluralities to majorities:

Perhaps that jump brought less-committed voters into Trump’s camp, and some of these supporters were too “shy” to admit they were voting for Trump, while he’d won only his core, unapologetic base in previous primaries.

But there’s also another plausible explanation: Trump was gaining support by the day during this period, and we’re including all polls in the three weeks before each election; many of the polls taken in the earlier part of that period drag down his average. You can see this in Indiana, where Trump’s average polling error in the two polls weighted most heavily by the FiveThirtyEight models (i.e., the most recent and/or historically accurate) had an average error of only a little more than 1 percentage point; the average error among all Indiana polls was nearly 10 percentage points.

OK, so what about the live-interview versus non-live-interview split? There isn’t any sign of a shy Trump-er effect in statewide polling when you split those up, either. Here are the 30 states where a live-interview poll was taken in the final three weeks before the election:

The breakdown is essentially the same as it was for the average of all polls. Trump underperformed his average live-interview poll 18 times (after taking into account undecideds). He outperformed just 12 times. In the median state, Trump underperformed his average poll by about 1 percentage point. In other words, it doesn’t appear that people were afraid to admit they planned to vote for Trump, or, at the very least, the shy Trump-er effect was canceled out by other factors such as his supporters perhaps being less likely to turn out.

In fact, live-interview polls were about as likely to overestimate Trump’s support as polls conducted using different technologies. Here’s the average polling error in the 29 states where non-live-interview polls were conducted within three weeks of voting or caucusing:

Trump did worse than the average non-live-interview poll in 16 states and did better in 13 states. As with the live-interview surveys, the average poll in the median state overestimated Trump by 1 percentage point. In other words, the live-interview and non-live-interview surveys had the same bias, on average. Additionally, Trump did better in the seven states leading up to and including Indiana than the average non-live-interview poll suggested he would, as was true for live-interview polls in states where they were taken. That’s not what we would expect if there were a shy Trump-er effect occurring in the later primaries.

My general findings here match those of The New York Times’ Nate Cohn, who looked at national polling rather than state polling. He found that Trump consistently did better in online polls (the method by which most non-live-interview national polls are conducted) than his cumulative vote percentage during the primary season. This holds when you control for the demographics of each state that voted. That is, the online national polls regularly overestimated Trump’s support among Republican voters.

Listen to the latest episode of the FiveThirtyEight politics podcast.

Does this mean there never was a shy Trump-er effect? Not necessarily. There could have been one that disappeared as the primary season progressed and voters realized that Trump was relatively popular among Republicans. If the polls are now underestimating Trump’s general electorate support because of a “shy Trump-er” effect, that could change in the next couple of months.

Indeed, there’s little reason to think that we’re in for a surprise in the general election because of Trump outperforming his polling on Election Day. As Cohn pointed out, the difference in Hillary Clinton’s margin over Trump between live-interview and online polls is fairly small right now at 2.5 percentage points, and that may have more to do with how the horse-race question is asked than the survey mode. This doesn’t mean that Trump won’t greatly outperform his polls or that people won’t be afraid to voice their support for him. But when you look at the evidence from the primary season and the general election survey results so far, I would bet on the polls getting Trump right again.