Emmanuel Macron’s 32-percentage-point victory in France’s presidential election runoff may end up being touted as a triumph for French pollsters, who consistently gave him a huge advantage. But it shouldn’t be. The polls leading up to the contest between the centrist Macron and his far-right opponent were the least predictive in French history, underestimating Macron’s support, rather than Marine Le Pen’s, to the surprise of some.
Here are a few takeaways.
A big, not-surprising miss
The average poll conducted in the final two weeks of the campaign gave Macron a far smaller lead (22 percentage points) than he ended up winning by (32 points), for a 10-point miss. In the eight previous presidential election runoffs, dating back to 1969, the average poll missed the margin between the first- and second-place finishers by only 3.9 points.
| MARGIN BETWEEN RUNOFF CANDIDATES | |||
|---|---|---|---|
| YEAR | TWO WEEK POLLING AVERAGE | ACTUAL RESULT | GAP |
| 2017 | +22.0 | +32.2 | 10.2 |
| 2012 | +7.1 | +3.3 | 3.8 |
| 2007 | +6.2 | +6.1 | 0.1 |
| 2002 | +56.0 | +64.4 | 8.4 |
| 1995 | +8.0 | +5.3 | 2.7 |
| 1988 | +10.0 | +8.0 | 2.0 |
| 1981 | -4.0 | +3.5 | 7.5 |
| 1974 | +2.8 | +1.6 | 1.1 |
| 1969 | +11.0 | +16.4 | 5.4 |
| Average before 2017 | 3.9 | ||
The previous largest error, 8.4 percentage points, occurred in 2002, when just a single public poll was conducted in the final two weeks of the campaign. This year, 18 surveys were conducted during the same period. Not one of them had Macron winning by as large a margin as he won by.
In fairness to French pollsters, even though the average margin in the final two weeks of polling was 22 points in Macron’s favor, polls conducted entirely in May, the final days of the campaign, found Macron’s lead to be an average of 23.9 points.1 But even those late polls undersold Macron’s victory.
All that said, even the biggest polling miss France has ever seen isn’t a totally shocking result. With a limited sample size of previous elections (as is often the case), there’s a decent chance that the true margin of error of the final polling averages will be underestimated (i.e., you underestimate how many results are in the tail ends of the distribution). Before this year, there had been only eight previous French runoff elections with polling during the final two weeks of the campaign. That’s why I noted on Friday that the true margin of error in the French runoff polling was likely somewhere between 10 and 12 percentage points even though there had never been a runoff in which the polling average had missed by that much.2
Please consider the margin
It’s unlikely that many people will remember this election in France as a big polling miss. The French polls had Macron winning, and he won. But that’s really the wrong way to look at these things. Most elections aren’t that close, so pollsters usually get the winner and loser right. If you’re focusing only on who won and who lost, you’ll end up with the impression that pollsters are far more accurate than they really are.
Perhaps that’s why people seem to think of the 2016 U.S. presidential election as such a black mark on pollsters’ records. But if pundits and analysts in the U.S. had been looking at how well polls in previous years had predicted the vote-share margin between the candidates, they would have been far less shocked by Donald Trump’s victory. As I wrote in the final week of that campaign, Trump was “just a normal polling error behind Clinton.” I based that analysis on how much the final polls had missed the margin between the top two candidates since 1968.
Indeed, looking at how accurately polls predict the final margin in elections gives you a much more accurate sense of their fallibility. In 2016, many analysts were going into Election Day thinking the polls had been basically perfect of late, identifying the winning in 2012, 2008 and 2004. Of course, they weren’t perfect. Polls in 2012, for instance, correctly showed President Obama winning re-election, but they undershot his margin of victory by about 3 percentage points. And they’ve been off by far more than that in previous years. So Trump’s victory in 2016 — he was down by about 4 points in national polls — shouldn’t have been surprising.
The difference between France in 2017 and the U.S. in 2016 is that the trailing candidate ended up winning in the U.S. And while that may make all the difference in the world in terms of who governs, focusing on who won is a terrible way to understand polls.
No one was that “shy”
Some analysts have argued that people are afraid to admit that they are voting for a far-right candidate such as Le Pen because they don’t want to give a socially undesirable response. That theory was bolstered twice last year, both when the “leave” vote in the U.K. referendum over leaving the European Union did slightly better than polls suggested and when Trump outperformed his polling. But the “shy insert-far-right-candidate here” theory doesn’t hold up when you look at a larger sample of European elections. And it didn’t hold up in France: There was no systematic bias in the polling against the far-right candidate (Le Pen).
As my colleague Nate Silver has pointed out, right-wing populist candidates and parties (in local and parliamentary elections) have, on average, pretty much matched their polling averages3 in European elections since 2012.
| DATE | COUNTRY | ELECTION | RIGHT-WING PARTY | POLL AVG. | ACTUAL |
|---|---|---|---|---|---|
| 5/7/17 | France | Presidential | National Front | 39.4% | 33.9% |
| 4/23/17 | France | Presidential | National Front | 22.3 | 21.3 |
| 4/9/17 | Finland | Municipal | Finns Party | 10.2 | 8.8 |
| 3/26/17 | Bulgaria | Parliamentary | United Patriots | 10.2 | 9.1 |
| 3/15/17 | Netherlands | General | Party for Freedom | 14.6 | 13.1 |
| 12/4/16 | Austria | Presidential | Freedom Party | 50.7 | 46.2 |
Indeed, the French presidential election is the sixth consecutive European election in which the populist right-wing candidate or party underperformed its polling. This includes Le Pen in the first round of France’s presidential election. Of the European elections that have involved a far-right wing candidate or party and have taken place since Trump won the White House in November 2016, the only candidate or party to win when polling projected a loss was left-wing Austrian presidential candidate Alexander Van der Bellen. (Van der Bellen beat the far-right, candidate Norbert Hofer.)
None of this is to say that there aren’t “shy voters” in the electorate. It’s just that we may be thinking about them in the wrong way. Instead of undercounting conservative support because people are afraid to give a socially undesirable response, the polls may simply be missing unenthusiastic supporters — people who aren’t excited about their candidate enough to answer a poll but still vote. In fact, when the idea of a “shy” voter was originally formed in 1992, it had nothing to do with right-wing populists. Instead, pollsters were underestimating the strength of the mainstream and relatively milquetoast Conservative Party in the U.K.
“Milquetoast,” in fact, has been used to describe Macron. In the 2017 French election, his voters were more likely to say that they were voting against Le Pen than for Macron. A Suffolk University poll also indicates that voters who liked neither candidate went overwhelmingly for Macron. In the 2016 U.S. presidential election, as well, Trump won because people who were unenthusiastic about both candidates (i.e., had an unfavorable view of both) went in large numbers for Trump. Maybe we should talk less about “shy” voters and more about “apathetic” voters or “reluctant” voters.