The post I put up early Thursday morning on Herman Cain prompted its share of annoyed reactions from some of the journalists and analysts whose work I admire. This is to be expected, since the article was essentially a critique of expert judgment.
It is, however, a fairly specific critique rather than a broad one. So let me restate its thesis more explicitly:
Experts have a poor understanding of uncertainty. Usually, this manifests itself in the form of overconfidence: experts underestimate the likelihood that their predictions might be wrong.
Examples of this can be found in numerous fields. Economics is an obvious one. In November 2007 — just a month before the economy officially went into recession — economists polled in the Survey of Professional Forecasters thought there was only about a 1 in 500 chance that economic growth would decline by 2 percent or more in 2008. (In fact, it declined by 3.3 percent). Many or perhaps most published findings in medical research are false — researchers cannot reproduce them when they try to re-create the experiment.
But political forecasts may be especially vulnerable to this. A long-term study of expert political forecasts by Philip E. Tetlock, a professor of psychology at the University of Pennsylvania, found that events that experts deemed to be absolutely impossible in fact occurred with some frequency. In some fields, these zero-percent-likelihood events came into being as much as 20 or 30 percent of the time. (What is an expert, by the way? Mr. Tetlock defines it as anyone who makes a living writing or thinking about politics.)
These are not semantic distinctions or errors around the margin — saying something has a 2 percent chance of occurring when really there is a 3 percent chance, or a 0.01 percent chance when really there is a 0.02 percent chance. Expert estimates of probability are often off by factors of hundreds or thousands.
Coming across these findings, as I have in the process of writing a book on prediction that is due to be published by Penguin Press next year, has had a somewhat radicalizing effect on my own views about forecasting. I used to be annoyed when the margin of error was high in a forecasting model that I might put together. Now I view it as perhaps the single most important piece of information that a forecaster provides. When we publish a forecast on FiveThirtyEight, I go to great lengths to document the uncertainty attached to it, even if the uncertainty is sufficiently large that the forecast won’t make for punchy headlines.
I could go on about this for pages and pages — and I will, in the book. But let’s bring the discussion back to Mr. Cain.
Ben Smith, in a blog post at Politico, perceived Thursday morning’s post as representing a juxtaposition of conventional wisdom against statistics — sort of a political version of Moneyball. Conventional wisdom holds that Mr. Cain is not a “serious” candidate; the polls say that he’s leading the Republican field. Conflict! Particularly if Mr. Cain wins, Aaron Sorkin could make a good movie about it.
But this is somewhat beside the point. I actually agree with the conventional wisdom that Mr. Cain is in much worse shape than his polling would imply. If you designed a forecast based solely on the current polling, Mr. Cain’s chances of winning the Republican nomination would work out to something like 40 percent. I think Mr. Cain’s chances are much lower than that. A toy model I have that accounts for both polling and nonpolling factors, including things like Mr. Cain’s lack of traditional experience or credentials, would put his chances at more like 10 or 15 percent. More objectively driven models that account for two major factors — polling and endorsements — would probably get you somewhere in the same range.
All of this, however, is speculative. These models are extremely prone to problems like data-dredging and overfitting — fancy ways of saying that they may mistake correlation for causation. Change a few assumptions, specify the model somewhat differently, and you may get a radically different result.
Mr. Smith and some other observers, however, are coming to their conclusions without any statistical model; instead it’s just a judgment call. “The Republican Party isn’t going to nominate the short-staffed former pizza executive and motivational speaker presently touring Alabama,” Mr. Smith writes. The forecasting literature suggests, however, that as badly as overfit statistical models can bungle estimates of uncertainty, our gut-feel sense for it is even worse.
One might expect it to be especially bad in the case of presidential primaries. There have been only about 15 competitive nomination contests since we began picking presidents this way in 1972. Some of them — like the nominations of George McGovern in 1972 and Jimmy Carter in 1976 — are dismissed by experts if their outcomes did not happen to agree with their paradigm of how presidents are chosen. (Another fundamental error: when you have such little data, you should almost never throw any of it out, and you should be especially wary of doing so when it happens to contradict your hypothesis.) One or two past precedents are mistaken for iron laws: Wesley Clark’s campaign did not go well; ergo, military commanders make for bad candidates.
And keep in mind that there are not even one or two precedents in Mr. Cain’s case; there are none. No past candidate has had his combination of strong polling and weak fundamentals. It is specious to focus on the latter condition while ignoring the former. Past “unconventional” candidates like Pat Buchanan, Pat Robertson, Jesse Jackson, Steve Forbes and Morry Taylor may not have done especially well — but this would have been predicted by their polling as well as by their fundamentals. (These candidates, in fact, generally performed about in line with their polling rather than underachieving it.) In Mr. Cain’s case, these numbers are in conflict.
In short, while I think the conventional wisdom is probably right about Mr. Cain, it is irresponsible not to account for the distinct and practical possibility (not the mere one-in-a-thousand or one-in-a-million chance) that it might be wrong. The data we have on presidential primaries is not very rich, but there is abundant evidence from other fields on the limitations of expert judgment.
In May, George F. Will said it was almost certain that either Tim Pawlenty or Mitch Daniels would win the Republican nomination. Mr. Will has gotten enough right over the years to have earned a mulligan or two. But experts who use terms like “never” and “certain” too often are playing Russian roulette with their reputations.