Skip to main content
ABC News
Despite Keys, Obama Is No Lock

Allan J. Lichtman, a presidential historian at American University, has issued a prediction that, given an economy still teetering on the brink of recession and President Obama’s 40 percent approval ratings, looks awfully bold.

“Even if I am being conservative, I don’t see how Obama can lose,” Mr. Lichtman told Paul Bedard of U.S. News & World Report.

Mr. Lichtman’s prediction is based on his 1984 book “The Keys to the White House”. The book cites 13 factors that can work for or against the party of the incumbent president. If at least eight of the 13 keys are scored in favor of the incumbent, he will win the election, Mr. Lichtman says; if he gets seven or fewer, he will not.

The book claims to have called the winner of the popular vote correctly in each election since 1860. (It would not have gotten the winner of the Electoral College right in 1876, 1888 or 2000, when the popular and electoral vote split.) That’s 38 elections in a row! Superficially quite impressive.

But ‘superficial’ is the crucial term here. There are several problems with this model, and its results should be taken with a grain of salt.

First has to do with the nature of the keys themselves, several of which are quite subjective.

1. Party mandate: After the midterm elections, the incumbent party holds more seats in the House of Representatives than it did after the previous midterm elections.
2. Contest: There is no serious contest for the incumbent party nomination.
3. Incumbency: The incumbent party’s candidate is the sitting president.
4. Third Party: There is no significant third party challenge.
5. Short-term economy: The economy is not in recession during the election campaign.
6. Long-term economy: Real per capita economic growth during the term equals or exceeds mean growth during the previous two terms.
7. Policy change: The incumbent administration effects major changes in national policy.
8. Social unrest: There is no sustained social unrest during the term.
9. Scandal: The incumbent administration is untainted by major scandal.
10. Foreign/military failure: The incumbent administration suffers no major failure in foreign or military affairs.
11. Foreign/military success: The incumbent administration achieves a major success in foreign or military affairs.
12. Incumbent charisma: The incumbent party’s candidate is charismatic or a national hero.
13. Challenger charisma: The challenging party’s candidate is not charismatic or a national hero.

This is most obvious in the case of the final two variables, which have to do with the charisma of the incumbent candidate and an opponent. I’m not of the view that there’s no such thing as “candidate quality,” or that charisma doesn’t matter. But it’s awfully easy to describe someone as charismatic when he or she is ahead in the polls — or when you have the advantage of hindsight and know who won an election.

Mr. Lichtman, for instance, scored Mr. Obama as charismatic in 2008, but not John McCain (even though Mr. McCain, with his service in the Navy, might have met Mr. Lichtman’s description of a “national hero”). However, Mr. Lichtman does not score Mr. Obama that way now.

At the same time, he scores some other debatable keys in favor of Mr. Obama. For instance, he gives him credit for having effected “major changes in national policy” because of the health care bill and stimulus package. Both those initiatives, especially the health care bill, are rather unpopular. On what premise will they increase his re-election chances?

Likewise, he does not call the war in Afghanistan a foreign policy failure, even though a majority of Americans think the war has gone badly and we should no longer be involved there. I’m not saying that Mr. Lichtman scored the issue wrongly — one can note that the United States does seem to have weakened Al Qaeda, or argue that any failure would be on President George W. Bush and not Mr. Obama. But it’s a debatable case.

The heart of the problem is something a little different, however: the formula is not actually all that accurate. Although it may have gotten the winners right, it does not do particularly well at accounting for their margin of victory.

Megan McArdle, a senior editor for The Atlantic, gets at this point by trying to apply the formula to Herbert Hoover, who lost to Franklin D. Roosevelt by 18 percentage points in 1932. She notes that between five and seven keys could easily be scored in favor of Hoover (Mr. Lichtman went with five) — short of the eight that Mr. Lichtman says are needed for re-election, but close enough that you wouldn’t have expected Hoover to lose in a landslide.

We can generalize this point by looking at the number of keys that each incumbent party candidate had and compare it to the margin of victory, or defeat:

As you can see, there is quite a bit of variance. For instance, James A. Garfield (in 1880), Calvin Coolidge (in 1924), Richard Nixon (in 1972) and George W. Bush (in 2004) were all scored as having nine keys in favor of them. All four won their elections. But Coolidge and Nixon won by 23 and 25 percentage points, respectively — while Garfield won the popular vote by less than 2,000 votes! Mr. Bush’s re-election in 2004, of course, was also quite close.

In fact, the 95 percent confidence interval on a projection using Mr. Lichtman’s formula is about plus or minus 16 points, which is certainly not especially precise. (Four elections in fact fall outside of the 95 percent confidence interval: 1912, 1920, 1924 and 1972.)

Looking at 1972 again, for instance, Mr. Lichtman’s formula would have predicted a Nixon victory by about 7 percentage points — when in fact, Nixon won by 23, a 17-point difference. Had George McGovern won the election by 10 points, the formula would have been just as accurate!

Or look at 1960: Nixon was scored as having just four keys in his favor, tied for the lowest score ever — but lost the popular vote by just one-tenth of a percentage point to John F. Kennedy. The formula was very, very close to being very, very wrong on this and other occasions.

Assuming that the errors in the formula are normally distributed, we can also use these historic results to translate the number of keys that an incumbent has into his probability of winning another term. For example, an incumbent who has nine keys scored in his favor — as Mr. Obama has now — has about a 79 percent chance of winning re-election, given the formula’s margin of error.

I’m sure the White House would be thrilled with a 79 percent chance of winning another term. But this hints at a another, more fundamental issue. As you can see from the table, there are lots of cases in which an incumbent’s re-election is an open question: one having eight keys scored in his favor, for example, should win about 65 percent of the time over the long run — but not 100 percent of the time. One having five keys scored in his favor, as Mr. Hoover did, should usually lose — but will pull out the victory about 18 percent of the time.

What are the odds of all these close calls just having happened to turn out right? Very low. Given the margin of error inherent in the formula, the chance of it having called the winner of the popular vote correctly in all 38 elections is only about 0.03 percent, or about 1 chance in 3,300.

In other words, given that the formula has done quite badly in predicting the margin of victory on several occasions (1932, 1960, 1972, etc.) it has been extremely lucky not to have gotten the winner wrong on at least one occasion. Over the long run, the formula should get about 20 percent of elections wrong, which would be about seven or eight misses out of a sample of 38.

Or … maybe it’s something other than luck. Although none of Mr. Lichtman’s keys are intrinsically ridiculous (for example, “which candidate had more ‘n’s in their name”), one can conceivably think of any number of other areas that might have been included in the formula but which are not — looking at how messy the primaries were for the opposition party, for example, or the inflation rate, or the ideological positioning of the candidates. (I mention these particular ones because there is some empirical evidence that they do matter.)

If there are, say, 25 keys that could defensibly be included in the model, and you can pick any set of 13 of them, that is a total of 5,200,300 possible combinations. It’s not hard to get a perfect score when you have that large a menu to pick from! Some of those combinations are going to do better than others just by chance alone.

In addition, as I mentioned, at least a couple of variables can credibly be scored in either direction for each election. That gives Mr. Lichtman even more flexibility. It’s less that he has discovered the right set of keys than that he’s a locksmith and can keep minting new keys until he happens to open all 38 doors.

By the way — many of these concerns also apply to models that use solely objective data, like economic variables. These models tell you something, but they are not nearly as accurate as claimed when held up to scrutiny. While you can’t manipulate economic variables — you can’t say that G.D.P. growth was 5 percent when the government said it was 2 percent, at least if anyone is paying attention — you can choose from among dozens of economic variables until you happen to find the ones that pick the lock.

These types of problems, which are technically known as overfitting and data dredging, are among the most important things you ought to learn about in a well-taught econometrics class — but many published economists and political scientists seem to ignore them when it comes to elections forecasting.

In short, be suspicious of results that seem too good to be true. I’m probably in the minority here, but if two interns applied to FiveThirtyEight, and one of them claimed to have a formula that predicted 33 of the last 38 elections correctly, and the other one said they had gotten all 38 right, I’d hire the first one without giving it a second thought — it’s far more likely that she understood the limitations of empirical and statistical analysis.

As for Mr. Obama, I’d note that only two of Mr. Lichtman’s 13 keys pertain to the economy, or 15 percent. Even our Republican readers, I think, would be ready to concede him another four years if the economy accounted for only 15 percent of voters’ decision. Mr. Lichtman’s keys do remind us that there are a number of noneconomic areas — for instance, the killing of Osama bin Laden, or the general avoidance of scandal — in which Mr. Obama has performed well.

But chances are that the economy is going to be much more important than that: my research suggests that it accounts for about half of a voter’s decision. If the economic factors are bad for Mr. Obama, and noneconomic factors are basically good for him, then that’s why the election figures to be on a razor’s edge.

Nate Silver founded and was the editor in chief of FiveThirtyEight.