Over at The Stump, Noam Scheiber tracked down a Pew Research study that purports to identify the presence of both a Bradley Effect and a Reverse Bradley Effect in the results to date in the Democratic primaries. For those of you who are not familiar with the Bradley Effect, the long and short of it is the idea that black candidates tend to systematically underperform their polls on election day, perhaps because people lie to interviewers in surveys so as to be politically correct. The Reverse Bradley Effect would be — well, just the reverse — a black candidate systematically overperforming his poll numbers for some or another reason.
The gist of the Pew study can be found in the chart below (larger version here), which looks at the extent to which Barack Obama overperformed or underperformed his polls in a number of primary states, and compares that to the African-American population in the state. The authors report that Obama has outperformed his polls in states with high African-American populations, and underperformed them in states with low ones. They find a correlation of .74, which is quite high — it would imply that more than half of the state-by-state polling errors are explained by the racial composition of that state alone.
I have a couple of issues with the way this study was conducted. Actually, just one issue, but it’s a pretty big one. The authors seem to have cherry-picked their states. They exclude states like Connecticut, Maryland, and New York that had closed primaries, but give no explanation as to why. They do not include Vermont, even though it appears to have met their standard of having three polls conducted in the week before the election. They don’t include Florida which, officially-sanctioned primary or no, would seem to be as useful as any other data point insofar as the Bradley Effect goes. They don’t include the caucus states of Iowa and Nevada (there is actually a decent argument for that, since caucuses are conducted in public rather than the privacy of a voting booth, but I would tend to be inclusive rather than exclusive when testing my own hypothesis). They do include the “outlier” of Wisconsin, but seem to be annoyed by it — as though it’s Wisconsin’s fault for not conforming to their hypothesis.
So, I attempted to recreate their analysis, pulling my own numbers from Pollster.com, and adding back in the blacklisted states. My imitation of their graph is below:
Putting these other states back in turns out not to make all that much difference; the correlation drops from .74 to .62. Still, we notice the presence of a few more states, like Vermont and Iowa, that don’t seem to fit the hypothesis.
You’ll notice I’ve done something else too, which is to color code the graph. The states in blue are those that we define as Southern (this includes Florida and Virginia, but not Maryland or Missouri), whereas those in red are the rest of the country. The claim for the Reverse Bradley Effect is really just based on the strong pull exerted by five states: Virginia, Alabama, South Carolina, Georgia, and Mississippi. All those states have high black populations, but they also have another thing in common, which is that they are all Southern.
But what if we look at other states that have relatively high black populations, but are not in the South? New York and Illinois have fairly substantial black populations — but the polls were spot-on in each of those locations. In Maryland — which I consider a Northern state, and which demographically has much more in common with other Northern states than anything in the South — Obama outperformed his polls, but only barely so. New Jersey has a relatively large black population and Obama slightly underperformed his polls there.
So instead of drawing one regression line, let’s draw two: one to represent the South and the other to represent the rest of the country.
Well, it looks like we are dealing with two completely different sets of behaviors. The relationship in the South is quite strong — just as strong as the Pew authors found originally. But there is virtually no relationship between race and Obama’s performance at the ballot booth elsewhere in the country — the slight correlation you see is nowhere near statistically significant. The polls overestimated Obama’s performance in New Hampshire and Massachusetts — but underestimated it in Vermont and Wisconsin. They were largely accurate in states like New York and Maryland that have substantial black populations. There is just nothing happening with the Bradley Effect outside of the South, at least so far as this data can tell us.
Also of note: my study identifies the presence of a Reverse Bradley Effect in the South (Obama outperforming his polls in states with higher percentages of black voters), but not the presence of the plain ol’ Bradley Effect in either the North or the South. At no point do my regression lines for either region run substantially above zero, which is the point at which Obama would begin to underperform his polls.
Put differently: there is nowhere in the country where we have reason to subtract from Obama’s poll numbers based on the Bradley Effect. (Yes, Obama has underperformed his polls in some “white” states — but he has overperformed them in others that are whiter than Kurt Rambis). On the other hand, there is one specific group of states where we might want to add to Obama’s polls based on the Reverse Bradley Effect, which are Southern states with high African-American populations.
If the Reverse Bradley Effect is real, what is its raison d’être? There is a fair amount of academic literature on the effects of the race of the interviewer on survey results. People can guess quite accurately the race of the person on the other end of the line, and they might respond differently depending on that perception. It is probably safe to assume that the majority of interviewers are white, and correctly perceived as white. A white voter might not want to tell a (presumably) white interviewer that they’re voting for the black guy. Or, a black voter might feel intimidated by a (presumably) white interviewer, and not want to tell him they’re voting for the black guy. These effects may be far more tangible in the South, in which race is a much more explicit consideration in everyday life. Three other quick comments on this:
1. If I had to guess, I would guess that black voters might be more likely not to want to reveal their true candidate choice than white voters. This is because I have noticed that a lot of black voters tend to be classified as ‘undecided’ in pre-primary surveys, which might indicate the hedging of bets.
2. As Scheiber and others have noted, there appears to be less of a Reverse Bradley Effect in polls conducted by agencies like Rasmussen, Survey USA and PPP that use automated calling scripts (robopolls). This requires further research — but if true it would tend to validate my hypothesis.
3. Something else that is worth mentioning: Americans tend to think that other people have more racial hangups than they claim to have themselves. This might be why there is a Reverse Bradley Effect, rather than a Bradley Effect. You don’t think you’re a racist — but you think the person on the other end of the line might be, and so you lie about your candidate choice so as not to offend them.
With all that having been said, I may be overselling the Reverse Bradley Effect. What we know is that in Southern states with large black populations, Obama has outperformed his polls by a statistically significant margin. But we’re just guessing at why this is the case. It could be because the pollster’s turnout models are screwed up. It could be because minority voters are more likely to use cellphones as their primary line, and therefore won’t make it into surveys. It could be because of something I call the Frontrunner Effect, which is that there is some tendency for candidates who are already ahead in the polls to run up the score on election day.
Nor do we know if we have identified a universal effect, or whether it’s something specific to Obama and Clinton. Perhaps Southern voters feel badly voting against Hillary Clinton, who claims some heritage in the region. But the same might not be true when Obama is matched against John McCain, whom Southerners tend to feel lukewarm about. Or, it could be that there is a Reverse Bradley Effect among Democrats but a real Bradley Effect among Republicans, and the two things will cancel one another out in the general election. If you had to bet against the spread, however, you might want to take the over on Obama’s numbers in states like North Carolina and Virginia.