It’s pretty rare that a pollster calls his own survey an “outlier.” But that’s exactly what happened last week after a Monmouth University poll showed an approximate three-way tie between Bernie Sanders, Elizabeth Warren and Joe Biden. Patrick Murray, director of the Monmouth University Polling Institute — an A-plus-rated pollster according to FiveThirtyEight — issued a statement describing his latest Democratic primary poll as an outlier that diverged from other recent polls of the race. (Indeed, there were quite a few national polls last week, and most of them continue to show Biden in front, with about 30 percent of the vote, and Sanders and Warren in the mid-to-high teens.)
But Murray doesn’t have any real reason to apologize. Outliers are a part of the business. In theory, 1 in 20 polls should fall outside the margin of error as a result of chance alone. One out of 20 might not sound like a lot, but by the time we get to the stretch run of the Democratic primary campaign in January, we’ll be getting literally dozens of new state and national polls every week. Inevitably, some of them are going to be outliers. Not to mention that the margin of error, which traditionally describes sampling error — what you get from surveying only a subset of voters rather than the whole population — is only one of several major sources of error in polls.
What should you do about these seeming outliers? If you’re a pollster, you should follow Monmouth’s lead and publish them!! In fact, printing the occasional expectations-defying result is a sign that a pollster is doing good and honest work. Plus, sometimes those “outliers” turn out to be right. Ann Selzer’s final poll of Iowa’s U.S. Senate race in 2014, which showed Republican Joni Ernst ahead by 7 percentage points over her Democratic opponent, might have looked like an outlier at the time, but it was the only one that came close to approximating her 8.5-point margin of victory there. The small handful of polls that showed Donald Trump leading in Pennsylvania in 2016 look pretty good too, even though most Pennsylanvia polls had Hillary Clinton leading.
In the long run, failure to publish results that pollsters presume to be outliers can yield far more embarrassment for the industry than the occasional funky-looking set of topline numbers. Suppressing outliers is a form of herding, a practice in which pollsters are influenced by other polls and strive to keep results within a narrow consensus. Herding makes polling averages less accurate, and it makes polling less objective. And more often than you’d think, it winds up being a case of the blind leading the blind. One recent example comes from Australia, where despite the Labor Party holding only a narrow and tenuous lead, pollsters declined to publish polls showing the conservatives narrowly ahead instead. The conservatives went on to a modest win, yielding a national controversy about polling that could have been avoided if the pollsters had trusted their numbers instead of the conventional wisdom.
About 99.99 percent of you reading this right now aren’t actually pollsters, though. So what’s my advice to you as news consumers when you encounter a poll that looks like an outlier?
To a first approximation, the best advice is to toss it into the average. Definitely do not assume that it’s the new normal. You don’t need to read dramatically headlined newspaper articles and watch breathless cable news segments about it. In a race with many polls, any one poll should rarely make all that much news. But you shouldn’t “throw out” the poll either. Instead, it should incrementally affect your priors. In the case of the Monmouth poll last week, for instance, you shouldn’t have assumed that the race had suddenly become a three-way tie, but you should have inched up your estimate of how well Sanders and Warren were doing compared with Biden.
For extra credit, pay attention to sample size. The Monmouth poll surveyed only 298 Democratic voters, which is small even by the standards of primary polls (which often survey fewer voters than general election polls do). Sample size is a complicated topic — as I mentioned, sampling error is only one source of polling error, and it’s not always the most important one. But as a rough rule of thumb, any poll with fewer than about 500 or 600 respondents is substantially more likely to have outlier-ish results because of sampling error than one that surveyed a larger number of voters. And polls with only 300 voters are especially likely to have issues.
So that’s the Polling 101 answer. When you see a poll that looks like an outlier, just throw it into the average. If you want, you can give some consideration to the sample size and the quality of the pollster.
But if you’ve read FiveThirtyEight for a while, you’ve probably heard that Polling 101 answer before. So I’m also going to give you the Polling 201 answer. But I want you to promise that you’ll abide by it fairly strictly, rather than interpret it too liberally. Pinky swear? OK, great. Then here goes:
If a poll shows a significant change in the race, you should tend to presume it’s an outlier unless it’s precipitated by a major news or campaign event.
Corollary: You should be much more open to the possibility that a poll reflects a real change if it’s among the first polls following a major news or campaign event.
What do I mean by a “major” news or campaign event? Some fairly specific types of things. When I made you pinky swear earlier, I was asking you to stick precisely to this list:
- Debates.
- Candidates entering or exiting the race, or clinching their nominations.
- Primary and caucus results (e.g., the Iowa caucuses occur and that has knockoff effects on the next set of states).
- The conventions.
- The announcement of vice presidential candidates.
- The final week of the campaign.
- Spectacular, blockbuster news events that dominate the news cycle for a week or more. (There generally are only one or two of these per campaign cycle, if that many.)
The first five examples are fairly straightforward. The party conventions, for instance, almost always produce bounces, which then fade over the course of a few weeks. Debates can also produce shifts, which can range from permanent to (more often) ephemeral; Kamala Harris’s bounce faded after the first presidential debate, for instance. Be careful with the fifth category, vice presidential selections, since not many VPs are true game-changers. But an outside-of-the-box pick — i.e., Sarah Palin in 2008 — can sometimes produce a polling shift.
The sixth category, the end of the campaign, is less well-known as a source of polling movement, but the final days of the campaign can produce sharp shifts in the polls as undecided voters finally settle upon a candidate and as supporters of candidates who look like they can’t win (say, a Libertarian who is polling at 4 percent) hold their noses and pick one of the major contenders. Often, especially in primaries, this movement occurs fairly late — within the final week of the campaign or even the final 24 to 48 hours (in which case it may occur too late to show up in polls). There’s no guarantee that undecided voters will evenly divide themselves between the major candidates; in Wisconsin in 2016, for example, voters who decided in the final few days went almost 2-1 for Trump over Clinton.
So you generally should pay more attention to polling movement in the final few days of the campaign. Frankly, this is the time when you should panic a bit if the polls are moving away from your candidate.1
Essentially all of the polling shifts so far in the Democratic primary fall into one of the first two categories. Look at the RealClearPolitics average over the past six months, and the major changes you see are as follows:
- A big bump for Biden after his entry to the race in late April, which faded over the course of several weeks.
- A decline for Sanders coinciding with Biden’s entry into the race.
- A big bounce for Harris, and a decline for Biden, after the first debate — both of which gradually reversed themselves.
- An additional modest decline for Harris, and a modest increase for Warren, after the second debate.
- A modest-sized bounce for Beto O’Rourke after he began his campaign.
- And a bump for Pete Buttigieg in late April and early May.
So almost all of the sudden polling movement for the Democrats has been associated with debates or candidates launching their campaigns. The major exception is Buttigieg’s relatively abrupt surge, which may have been partly triggered by his town hall on CNN — certainly not an event that comes anywhere near qualifying under my seven categories above. Sharp polling movement sometimes does occur outside of these categories, but not very often. So you should err on the side of being conservative. It’s not that other sorts of news or campaign events can’t surprise you and change the polls. It’s just that you’d want to see several polls pointing toward a shift before you buy that they do.
The final category, blockbuster news stories, is the one where there’s the most room for subjectivity – and therefore the one you need to be most cautious about. Keep in mind that the overwhelming majority of news stories are less important to the campaign than they seem at the time. So if you’re the type of person whose life is caught up in the daily news cycle — or someone who works in politics for a living — you’re probably better off just ignoring this category entirely.
But stories that dominate the news cycle for a week or more and interrupt all other political coverage can change the polls, of course. To get a more objective idea of which stories qualify, you can look toward political aggregators like Memorandum, which archive their results to show which stories were dominating the news cycle on any given day. Many stories that people think of as political blockbusters really only last for two to three days.
What stories meet this threshold? In the 2018 midterms, probably only Brett Kavanaugh’s confirmation. In 2016, the “Access Hollywood” tape and the Comey letter. In 2012, nothing, really. In 2008, the financial crisis was an ongoing story, but the Lehman Brothers bankruptcy on Sept. 15 touched off a series of acute events that reoriented the race. Events such as the publication of the Mueller report, Hurricane Katrina, the killing of Osama Bin Laden, the stock market crash of 1987 and the start of the Iraq War would also have qualified, had they occurred in the middle of election campaigns.
Let’s close with one more reason not to get all that excited about short-term polling swings. It’s one I’ve already alluded a couple times, but it probably can’t be emphasized enough. Polling shifts driven by campaign and news events often reverse themselves once the news cycle moves on to another topic. So even if the movement is real, it may be temporary. It will be highly relevant if the election is right around the corner, but less so if it’s several months away.
By contrast, gradual, long-term polling movement — of the sort that Warren has benefited from over the course of several months, for example — can be more durable. It’s entirely plausible that Warren has been gaining a point or two every few weeks not because of any specific news stories but just as a result of persuasion as voters become more familiar with her campaign. If you’re a Warren fan, that’s what should get you excited — and not the next outlier poll that comes along.