How The Giant Democratic Primary Field Messes With Polls

There are already a lot of things that a smart poll-watcher has to account for when comparing surveys. Were they done by the same pollster? Did they query the same type of population (likely voters, registered voters, all adults)? To this list, the volatile 2020 Democratic primary field has added one more: Do they ask about the same candidates?

With upwards of 20 candidates and potential candidates, pollsters are forced to make editorial decisions about whom to ask about. And with new candidates still popping up out of nowhere, the field is enough in flux that it looks different every month. By our count, nine pollsters have polled either the same state or the entire nation more than once since the 2018 midterm elections, which is when we started collecting 2020 primary polls. But none of these pollsters have kept the same list of candidates for every poll — although some have been more consistent than others. This means great care must be taken in declaring a candidate is surging — or struggling — even when comparing his numbers in polls from the same pollster.Morning Consult polling when judging how big of a bump candidates get from their campaign kickoffs. In addition to polling thousands of Democrats every week, Morning Consult is conservative about changing the names it asks about. In the last seven weeks, it has only altered its candidate roster once, to reflect the fact that former New York City Mayor Michael Bloomberg, Ohio Sen. Sherrod Brown and former Attorney General Eric Holder announced they would not run.

“The candidates in the survey make a material difference,” Mike Greenfield, co-founder and CEO of the pollster Change Research, told FiveThirtyEight in an email. For example, when Change Research polled the nation in December, Joe Biden received 21 percent support, and Bernie Sanders received 16 percent. But when they polled the country again in early March, Biden had jumped to 36 percent and Sanders sat at 24 percent. But we can’t confidently say based on those two polls alone that Biden and Sanders had actually gained ground with the electorate. Why? The March poll did not give respondents the option to choose eight politicians who appeared in the December poll² and had combined for 17 percentage points, substituting them with 10 new names of late-breaking candidates, like South Bend, Indiana, Mayor Pete Buttigieg and Washington Gov. Jay Inslee. Many voters whose first choice had dropped off the list switched to Biden and Sanders in the latter poll, Greenfield said. However, “even with that challenge, you can still tease out some very meaningful changes.” For example, Greenfield said that Beto O’Rourke’s dip in support from 21 percent in December to 7 percent in March can’t be accounted for by different candidate rosters alone.

Likewise, between two David Binder Research polls of Iowa (also in December and March), it’s unclear which candidates’ polling bumps were organic and how much of a bump or decline was due to respondents being offered different choices. The March poll did not ask about four candidates who totaled 7 percentage points combined in December; meanwhile, the December poll didn’t ask about 12 candidates who combined for 12 points in March.

The good news is that, in most polls, the candidates who are rotated in or out are polling poorly, so it doesn’t make a huge difference whether they’re included or excluded. Plus, there can be good reasons for a pollster to update the roster of candidates it asks about; it’s important that a poll reflect the most up-to-date state of the race (e.g., Bloomberg looks like less of a factor now than he did in December, and Buttigieg looks like more of one).

This is especially true since getting at least 1 percent support in three national polls is one of two ways a candidate can qualify for the first two Democratic primary debates (the other is through fundraising). But probably unsurprisingly, the candidates being left out of polls most often are those who most need the exposure that a debate would bring:

The fact that poll results influence which candidates can get into the debates weighs on some pollsters when they decide how many candidates to ask about. For instance, the Monmouth University Polling Institute wrote in a statement that it “will attempt to be as inclusive as possible in its polling of the Democratic field in these early stages of the campaign.” And Jeff Cartwright, a vice president at Morning Consult, told FiveThirtyEight in an interview that his company also tries to be “as expansive as possible.” Despite fears that long surveys may overwhelm respondents, the numbers show that most pollsters err on the side of inclusiveness. The median Democratic primary polling question in the FiveThirtyEight database has asked about 16 candidates, and one even asked about 26 different candidates:

Collecting polling data on the large number of candidates in the 2020 Democratic primary can be especially tricky for pollsters who rely on automated phone polls, where a respondent is asked to “press 1” for Jane Candidate and “press 2” for Paula Titian. These pollsters are naturally limited to 10 choices because there are only 10 numbers on a phone. (Emerson College has experimented with a couple ways of getting around this, but it led to wide variations in its results.)

And even online and live-caller pollsters have to draw a line somewhere. (There are more than 200 Democrats officially running for president, and it’s infeasible to ask about all of them.) Cartwright said Morning Consult looks at active candidates who have some level of name recognition and pays close attention to news reports to determine who are the most likely potential candidates. Sometimes the pollster will conduct test polls on the side to see if a new candidate has gained enough traction to be included in its main poll. And according to Monmouth’s statement, in addition to declared candidates who have held federal or statewide office, the pollster will also include those “who have shown fundraising ability … as well as those who may have the potential to register support in early states based on other metrics.”

In sum, the fluid field is one reason not to read too much into early 2020 polls. When comparing polls — even when they’re from the same pollster — take care that the candidate rosters don’t differ too much. And the next time you see a surprising result, make sure you scan the list of candidates to make sure no big-name potential candidates are unduly included or excluded. Overall, the most useful 2020 polls are those that come as close as possible to the final list of candidates that will appear on ballots in less than a year’s time. Of course, the trick is guessing right.