We’ve just launched FiveThirtyEight’s 2016 general election forecast, which projects how the 538 Electoral College votes could break down in the presidential election. The forecast will be continually updated through Election Day on Nov. 8. Here’s a bullet-point-style look at how it was built.
What’s new in the model since 2012?
- Not that much! It’s mostly the same model as the one we used to successfully forecast the 2008 and 2012 elections.
- There’s no special variable for Republican Donald Trump or Democrat Hillary Clinton. They’re treated the same as any other candidates would be with the same polling numbers.
- We built procedures to handle Libertarian Gary Johnson and other third-party candidates.
- We double-checked lots of assumptions and code.
- We’re now showing different versions of the model: the polls-only and polls-plus forecasts, and the now-cast (what would happen in an election held today).
Major themes and findings
- Think probabilistically. Our probabilities are based on the historical accuracy of election polls since 1972. When we say a candidate has a 30 percent chance of winning despite being down in the polls, we’re not just covering our butts. Those estimates reflect the historical uncertainty in polling.
- State polls > national polls. All versions of our models gain more information from state polls than from national polls.
- Errors are correlated. But while the election is contested at the state level, the error is correlated from state to state. If a candidate beats his polls to win Ohio, there’s a good chance he’ll also do so in Pennsylvania.
- Be conservative early and aggressive late. Fluctuations in the polls in the summer are often statistical noise or short-term bounces. The model is trained to be conservative in reacting to them. Fluctuations late in the race are more meaningful, and the model will be more aggressive.
Three versions of the model
- Polls-plus: Combines polls with an economic index. Since the economic index implies that this election should be a tossup, it assumes the race will tighten somewhat.
- Polls-only: A simpler, what-you-see-is-what-you-get version of the model. It assumes current polls reflect the best forecast for November, although with a lot of uncertainty.
- Now-cast: A projection of what would happen in a hypothetical election held today. Much more aggressive than the other models.
Differences between polls-plus and polls-only
- Polls-plus combines polls with an economic index; polls-only does not.
- Polls-plus will include a convention bounce adjustment; polls-only will not.
- Polls-plus starts by assuming that likely voter polls are better for Republicans; polls-only makes no such assumption. Both models revise this assumption as more data becomes available.
- Polls-plus subtracts points from third-party candidates early in the race, while polls-only does not.
- Both models employ a regression that is based on demographics and past voting history. But polls-only weights the regression less and places less emphasis on past voting history.
- Polls-only accounts for more uncertainty than polls-plus.
- Polls-plus and polls-only will tend to converge as the election approaches.
Differences between polls-only and now-cast
- The now-cast is basically the polls-only model, except that we lie to our computer and tell it the election is today.
- As a result, the now-cast is very aggressive. It’s much more confident than polls-plus or polls-only; it weights recent polls more heavily and is more aggressive in calculating a trend line.
- There could be some big differences around the conventions. The polls-only and polls-plus models discount polls taken just after the conventions, whereas the now-cast will work to quickly capture the convention bounce.
Four major steps
All versions of the model proceed through four major steps:
- Step 1: Collect, weight and average polls.
- Step 2: Adjust polls.
- Step 3: Combine polls with demographic and (in the case of polls-plus) economic data.
- Step 4: Account for uncertainty and simulate the election thousands of times.
Step 1: Collect polls
Almost all state and national polls are included. If you don’t see a poll, it’s for one of these reasons:
Which poll version do we use?
Sometimes, there are multiple versions of a poll. For example, results are listed among both likely voters and registered voters.
- We prioritize polls as follows: likely voters > registered voters > all adults.
- If there are versions with and without Gary Johnson, we use the version with Johnson.
- All other ambiguous cases are considered ties. Sometimes, for instance, a pollster will publish results showing two likely voter models instead of one. Our program will average any such instances together.
Calculating a weighted average
We calculate a weighted average in each state, where poll weights are based on three factors:
Step 2: Adjust polls
There are five adjustments, listed here in the order in which the model applies them. (The trend line and house effects adjustments are generally the most important ones.)
- Likely voter adjustment
- Convention bounce adjustment (in only the polls-plus model)
- Omitted third-party candidate adjustment
- Trend line adjustment
- House effects adjustment
Likely voter adjustment
- Polls of registered voters and adults are adjusted to be equivalent to likely voter polls.
- The adjustment begins with a default setting but changes as the model collects data on polls that list both registered and likely voter numbers.
- Historically, Republicans gain slightly in likely voter polls — a net of 1 to 2 percentage points — compared with registered voter or adult polls. Therefore, in the polls-plus model, the default is that likely voter polls slightly favor Republicans. The polls-only and now-cast models ignore this historical precedent and use a default of zero.
- But so far this year, Trump isn’t gaining ground on Clinton in likely voter polls. In several polls, Clinton has done slightly better in the likely voter version, in fact. Thus, this adjustment doesn’t have much effect right now.
- Likely voter polls tend to show fewer undecided voters.
Convention bounce adjustment
- Historically, parties receive large but fleeting bounces in the polls after their party convention. For instance, Walter Mondale led Ronald Reagan 48-46 in one poll conducted just after the Democratic National Convention in 1984!
- The bounces have been smaller in recent years, but candidates can still come out “hot” after their conventions (e.g., McCain/Palin in 2008).
- The polls-plus model applies a convention bounce adjustment, subtracting points from a candidate’s polls just after his or her convention.
- Polls-only and now-cast do not apply an adjustment.
- As another line of defense, both polls-plus and polls-only weight polls less if they’re conducted in the immediate aftermath of the convention (but now-cast weights them fully).
- Polls-plus assumes that a modern-day convention bounce is worth 3 to 4 percentage points. But because the conventions occur back-to-back this year, the bounces could obscure each other.
Omitted third-party adjustment
- Also known as the “missing Johnson adjustment.” Because our default is to use polls with Johnson, we adjust polls that don’t list him.
- The model estimates how much of Johnson’s support comes from the major-party candidates, instead of from undecided voters. (Answer: Relatively little support for Johnson is from undecided. Clinton and Trump both poll considerably lower in polls that include Johnson.)
- The adjustment assumes Johnson takes his support from Clinton and Trump equally.
- The adjustment differs in each state. It will take more points away from Clinton and Trump in states it perceives to be good for Johnson.
Trend line adjustment
House effects adjustment
- House effects are persistent partisan “leans” in polls. For instance, Rasmussen Reports polls are typically Republican-leaning, relative to other polls.
- The model detects each polling firm’s house effect by comparing its polls to others of the same state.
- The model then subtracts a proportion of the house effect back out. The proportion depends on the number of polls each firm has conducted. For instance, say a pollster has a 3 percentage point Clinton-leaning house effect. The model might subtract only 1 point from Clinton if the firm has conducted only a few polls. But it might subtract 2.8 points if the firm has conducted dozens of polls, and the model had a very strong idea of its house effect.
- House effects are calculated separately for Clinton, Trump and Johnson. A pollster could have both a pro-Clinton and pro-Trump house effect if it tended to show few undecided voters, for instance.
- In calculating house effects, the model needs to determine what an average poll is, as a basis for adjusting the other polls. This average is weighted, based on each firm’s pollster rating. In other words, high-quality polls have more say in the house effects adjustment.
Step 3: Combine polls with other data
Adjusting the third-party vote
- Historically, third-party candidates tend to underperform their early polls. Essentially, some third-party voters may really be undecided voters using the third-party candidate as a placeholder. (Note that third-party candidates do not necessarily underperform their late polls. This is more of a concern in the summer and early fall.)
- Therefore, early in the race, the polls-plus model will subtract some of the vote from the third-party candidate based on this pattern and reallocate it to undecided.
- The polls-only model and the now-cast do not do this. They leave the third-party vote as-is.
Allocating undecided voters
- Undecided voters are split evenly between the major-party candidates. Empirically, an even split works better for presidential races than a proportional split.
- Late in the race, the third-party candidate will also get a share of undecideds.
- A small portion of the vote is also reserved for “other” candidates (e.g., Green Party candidate Jill Stein, etc.) in states where we expect four or more candidates to be on the ballot.
Projecting the national popular vote
- In all versions of the model, the national popular vote is held constant when combining polls with demographic data.
- For example, if Clinton is up by 5.1 percentage points nationally before the demographic regressions are applied, she’ll also be up 5.1 points after they’re applied.
- How do we project the national popular vote? There are two possible approaches: Top-down, using national polls, and bottom-up, estimating the national popular vote from state polls.
- The model uses a blend of both approaches but puts considerably more weight on the state polls strategy, which has been more accurate historically.
- In calculating the bottom-up estimate, the model controls for each state’s partisan voter index (PVI), a measure of how it voted in the past two presidential elections. Thus, it won’t be thrown off if we have lots of polling from blue states but little from red states, or vice versa.
National polls versus state polls
To recap, the model mostly uses state polls. But national polls can influence the forecast in some subtle ways:
- They’re helpful for calculating adjustments to the polls, especially the trend line adjustment and house effects adjustment.
- They’re used, in conjunction with the state polls, in estimating the national popular vote.
Partisan voter index (PVI)
Calculating demographic regressions
Instead of using one regression model, we take three strategies, which range from more simple to complex, and blend them together. The reason for this is that the more complex methods (especially strategy 3) are subject to potential overfitting. Hedging the complicated methods with simpler methods produces a better result.
Blending polls and regression
- The adjusted polling average in each state is combined with the regression. The regression estimate gets more weight early in the race and when there’s less polling. The regression gets 100 percent of the weight when there’s no polling in a state. The polling average can get as much as 95 percent of the weight late in the race in a state with abundant polling.
- Polls-only and now-cast give slightly less weight to the regression than polls-plus does.
- As a final step, the regression is recalibrated so that the overall national popular vote is unchanged. If a candidate gains ground in one state because of the regression, the model will necessarily have her lose ground in another.
- In other words, the purpose of the regression models is not to say the country’s demographics inherently favor Trump or Clinton. Instead, it’s to create a more realistic distribution of the projected vote across each state, especially in states with limited polling. We don’t want to have Clinton winning Kansas based on a single poll there, for instance, while she’s badly losing Nebraska.
Calculating the economic index
The “fundamentals” forecast
Blending polls and fundamentals
State elasticity scores
Step 4: Simulate the election
- The final major step is to account for the uncertainty in the forecast and simulate the election.
- The uncertainty decreases as Election Day approaches.
- The error from state to state is correlated. If Trump significantly beats his polls in Ohio, he’ll probably do so in Pennsylvania also. Figuring out how to account for these correlations is tricky, but you shouldn’t put too much stock in models that don’t attempt to do so. They’ll underestimate the chances for the trailing candidate if they assume that states are independent from one another.
Three types of error
Each simulation accounts for three potential types of error and uncertainty:
- National error. The polls are systematically off throughout the country.
- Demographic and regional error. The polls are off in states that have demographic or geographic factors in common.
- State-specific error. The polls are off in a particular state, with no effect on other states.
- In each simulation, a random number is drawn to model national error. It’s applied to every state about equally, subject to that state’s elasticity score.
- The magnitude of the national error is based upon: The amount of time until the election (more time = more error); the number of undecided voters (more undecideds = more error); and the number of third-party voters (more third-party votes = more error).
- Because there’s a significant undecided and third-party vote, national uncertainty is higher than usual this year.
Demographic and regional error
- Some states’ outcomes are more correlated than others. For instance, if Trump beats his polls in Minnesota, he’ll probably also do so in Wisconsin. But that might not tell us much about Alabama.
- The model simulates this by randomly varying the vote among demographic groups and regions. In one simulation, for instance, it might have Trump beating his polls throughout the Northeast. Therefore, he wins Maine, New Hampshire and New Jersey. In another simulation, Clinton does especially well among Hispanics and wins both Arizona and Florida despite losing Ohio.
- The simulations are conducted from a file of more than 100,000 voters, built from the exit polls and CCES.
- The following characteristics are considered in the simulations: religion (Catholic, mainline Protestant, evangelical, Mormon, other, none); race (white, black, Hispanic, Asian, other); region (Northeast, South, Midwest, West); party (Democrat, Republican, independent); and education (college graduate or not).
- To get a better sense of how this works, here’s a correlation matrix drawn from recent simulations. You can see the high correlation between Wisconsin and Minnesota, for example.
- Finally, the model randomly adds additional error specific to one state at a time.
- A state has more state-specific error when it has fewer polls.
- It also has more error when polls and demographics disagree. If the regression models and the adjusted polling average show significantly different results — such as in a state like Utah, for instance — that contributes to uncertainty.
- And states have more state-specific error when they have smaller populations. Small states usually have more demographic idiosyncrasies than larger ones; that makes them harder to poll and harder to model based on patterns we see elsewhere in the country.
Odds and ends
That’s basically it! But we’ll conclude with a few odds and ends:
- We usually run at least 20,000 simulations for each version of our model each day. That’s a lot, but it still produces a small amount of sampling error. You shouldn’t worry too much when win probabilities change by less than a percentage point.
- We simulate the vote by congressional district in Maine and Nebraska, which award one electoral vote to the winner of each district and two electoral votes statewide. Where available, district-level polling is used in these forecasts.
- If no candidate receives a majority of electoral votes, the model assigns the election to Trump half the time (because Republicans are very likely to control a majority of congressional delegations if the election is close) and to the winner of the popular vote the other half of the time.
Our distributions have fat tails
Handling the third-party vote
Third- and fourth-party ballot access
Tipping-point chance and voter power index
- Tipping-point chance is the probability that a state will provide the decisive vote in the Electoral College.
- More precisely, tipping-point chance is derived from our simulations, in cases where the popular vote is close enough that the Electoral College matters. In each simulation, the model sorts the states by Trump’s projected margin of victory or defeat there, from most favorable for Trump to most favorable for Clinton. It keeps adding up electoral votes for Trump until he gets to 270. The state that puts Trump over the top is the tipping-point state for that simulation. Two states share the tipping-point designation in the event of a 269-269 tie.
- Voter power index — what we called the “return on investment” index in 2012 — reflects the relative chance that an individual voter will cast the ballot that leads to the decisive electoral vote. If a state has a voter power index of 3.5, that means a vote there is 3.5 times more powerful than the national average.
- Voter power index is calculated by taking each state’s tipping-point chance and dividing it by the share of the national turnout we expect that state to represent.
- Voter power index tends to favor less populous states because they have a larger number of electoral votes relative to their populations. As of the 2010 Census, California had one electoral vote per 677,000 people, while Wyoming had one per 190,000 people.