Does the world really need another polling average?
Well, sure. Actually, we think having a variety of polling averages matters a lot in the presidential primaries — and the one you look at can change how you view the race. So we’ve just launched our national polling average for the 2020 Democratic primary, as well as one for every state where there’s an adequate amount of polling. Here’s Iowa, for instance:
Here’s South Carolina:
Here’s Nevada. Here’s California. We’d encourage you to click around a bit and then come back here once you’ve gotten a sense for how the numbers look.
Constructing a polling average is never quite so straightforward as it seems, but that’s doubly true in a primary campaign. Since turnout is relatively low compared to a general election, polls can differ a lot from one another given their assumptions about who’s going to vote. Public opinion can change quickly during the primaries; unlike in the general election, where the large majority of voters can reliably be expected to vote for one party or the other, primary voters are usually considering multiple candidates, so the overall process is a lot more fluid. Having a polling average that moves too slowly can be a big problem, as can having one that overreacts to every new poll. The pace of polling can be irregular — sometimes you’ll get several high-quality polls in a day, and sometimes you’ll go a couple of weeks without any. All of these factors make the methodological choices behind a polling average more important.
FiveThirtyEight Politics Podcast: Democratic primary, according to the early states
So here’s a description of our version of a polling average: a relatively brief overview of the features that make FiveThirtyEight’s primary polling averages different (and, we hope, slightly more accurate) than the alternatives. I’m keeping this relatively brief for two reasons: First, none of this differs that much from the polling averages we’ve constructed in the past for general elections, and second, these polling averages will soon be followed by our full-fledged primary forecast, which uses these averages as an “ingredient” but also considers a host of other factors. We’ll save the discussion of the forecast for later, though. For now, listed in rough order of how much they can affect our averages, here are the five key things that make our polling averages a little different:
Differentiator 1: We adjust state polls based on trends in national polls
A hallmark of our general election forecasts, dating all the way back to our first versions in 2008, is what we call a “trend line adjustment.” Basically, in states that haven’t been polled recently, we make inferences about what’s going on there using national polls or polls from other states that have been surveyed recently. If President Trump gained 3 percentage points in national polls, for instance, but North Carolina hadn’t been polled recently, you could probably infer that he’d also gained 3 points, or thereabouts, in North Carolina.
We apply this adjustment for a good reason: Trend-line-adjusted averages have been quite a bit more accurate, historically. That is, once someone does get around to polling North Carolina, it usually turns out that Trump did gain about 3 points. Trend-line-adjusted averages have also been insightful so far this cycle — they anticipated, for instance, that Sen. Elizabeth Warren’s numbers would decline in state polls in November and early December once she began to slump in national polls.
So we’re now applying a slightly simplified version1 of the trend line adjustment to the primaries. (Note that all polling averages you see on our state polling pages reflect this trend line adjustment.) Say that Sen. Cory Booker has surged by 5 points in national polls, for instance, but we haven’t seen a recent poll in Nevada. Our average would assume that he’d also improved his standing by 5 points in Nevada, other things held equal.
There are a few complications: For instance, the adjustment is nonlinear, which can have meaningful effects if a candidate is polling in the low single digits. But in short, the trend line adjustment can have fairly large effects if a state hasn’t been polled much recently. It doesn’t affect our numbers much, conversely, if there are a lot of recent, high-quality polls from that state.
Differentiator 2: We adjust for house effects
“House effects” are when certain pollsters consistently show better results for certain candidates. Emerson College, for instance, has usually shown optimistic results for Sen. Bernie Sanders in polls it has published so far in the primary campaign, while Morning Consult’s polls tend to have pretty good numbers for former Vice President Joe Biden.
All of FiveThirtyEight’s general election polling averages adjust for house effects, and we’re now doing the same for our primary averages. In fact, we found that the house effects adjustments we’ve used in the past were slightly too consevative for the primaries, so they’ll be a bit more aggressive this year.2
House effects are calculated for each candidate separately. So, for instance, Morning Consult has a Biden house effect adjustment, a Pete Buttigieg adjustment, a Tom Steyer adjustment, and so on. National polls can influence the house effects adjustment in the states and vice versa, and polling in one state can influence the house effects adjustment in other states.
Differentiator 3: Our average adjusts more quickly after major events
We’ve long recommended that you should consider news events when determining whether a polling shift is signal or noise. If Trump literally did shoot someone on Fifth Avenue, or the Martians invaded Washington and he valiantly fought them off, it wouldn’t be surprising if there were a sharp shift in his approval rating. Conversely, if we were in the midst of a boring news cycle where nothing much was happening, a poll showing a big swing in his numbers would be more likely to be an outlier.
We’re now applying this sort of logic to our primary polling average, though in a much more formal and rigorous way. While we aren’t expecting a Martian invasion, there are certain types of events in the primaries that predictably can have large effects on the polls (i.e. they have historically). Specifically, these include, in order of importance:
- The outcomes of primaries and caucuses (e.g. a candidate can get a bounce after Iowa or Super Tuesday)
- Major candidates entering or exiting the race
- Debates
Following these three types of events, our polling average will be more aggressive about deeming swings in the polling average to be signal rather than noise. As a corollary, it will be less aggressive when there are apparent polling shifts that aren’t precipitated by one of these events. We’ll revisit this in future articles, but note that the importance the model assigns to events works on a sliding scale. No offense to our Guamanian readers, for instance, but, historically, Iowa or Super Tuesday tends to move the polls a lot more than the Guam caucus — and our averages reflect this.
Differentiator 4: We’ve carefully set our average so it doesn’t move too fast or too slow
Over the years, we’ve found that there’s no particular default set of assumptions that will give you a good polling average in every circumstance. Applying the aggressive settings from our presidential approval rating average to our generic ballot tracker makes the generic ballot much too “bouncy,” for instance. (We learned that one the hard way.) Conversely, applying the conservative settings from our generic ballot tracker would make our presidential approval rating average too sluggish to pick up on real swings in Trump’s numbers.
So we think the only good way to determine the “right” settings for a polling average is to do it empirically. There are a couple of ways that you could do this:
- You can tune the settings so that they optimally predict future polls. That is, if our approval rating average has Trump at 42 percent, that means 42 percent is our best guess for what a new Trump approval rating poll would say.
- Highly related to the above, you can tune the settings to minimize autocorrelation. That is to say, the current polling average should reflect all information about the current state of the polls and your average shouldn’t predictably move upward or downward from that point in time. For example, if Sanders improves from 15 percent to 17 percent in your polling average, he should be equally likely to continue gaining ground (improve beyond 17 percent) or to revert to where he was before (decline from 17 percent) in future editions of your polling average.
- For polling sequences that culminate in an election, like the New Hampshire primary poll average, you can test how accurately the polling average predicts the eventual election result.
The settings we chose for our primary polling averages are designed to optimize these qualities based on our historical database of primary polls since 1972. In general, it’s appropriate to apply relatively aggressive settings in the primaries as compared to the general election, as the former tend to be much more dynamic than the latter due to the lack of partisan guardrails.
I’ll refrain (for now) from going into more detail on exactly what these parameters are and how we’ve set them. (There are actually quite a few parameters, ranging from how you trade off recency versus having a larger sample of polls to which kernel density function to apply.) As a matter of practice, though, the FiveThirtyEight polling average represents something of a compromise between the RealClearPolitics approach of averaging recent polls and The Economist’s technique of drawing a trend line.
Differentiator 5: We use objective criteria to decide which polls to include
For many reasons, we prefer to avoid having to make any ad hoc decisions about which polls to include in our average. So our approach has always been to include almost all polls but to weight them based on our pollster ratings (which in turn reflect a combination of how accurate the pollster has been historically and the methodology it uses) and the polls’ sample size. We’re applying this long-standing process to our primary polling averages as well.
Note that I said “almost all polls” rather than “all polls” because there are some rare exceptions. We don’t include polls from firms that are banned by FiveThirtyEight because we suspect them of having faked data. And for the primaries, we won’t be including internal polls that are released to the public by one of the campaigns,3 or surveys that test super hypothetical matchups, such as a head-to-head poll conducted when more than two candidates are still running.
And that’s about it. Again, we’ll have much more detail on some of this when we launch our forecast. But in the meantime, please go click around and see how the race looks nationally and in the various states. Is Buttigieg losing steam in surveys of Iowa? Has Warren arrested her decline in national polls? Now you can decide for yourself.