Perhaps now that the election is over and things are slightly calmer around here, we can take questions from readers a little more often — or even do a “mailbag” column once in a while.
I got an e-mail from Richard S. that gets at a distinction that is key to understanding Thursday’s column on the performance of various polling firms in Tuesday’s elections, in which I referred to the “bias” present in the results obtained by certain pollsters (particularly Rasmussen Reports). He writes:
I’ve been following 538 for a long time, and I was curious about your change in terminology from ‘house effect’ to ‘bias.’ You have always been very careful in previous posts to use the former term, often stressing that the house effect was a result of methodology and did not necessarily reflect any intentional bias on the part of the pollster. In your post today you repeatedly used the term ‘bias’ instead.
I understand that in statistics ‘bias’ is the correct term for such systematic errors in data collection, but when discussing opinion polling ‘house effect’ seems to be the more common term; and I fear that for lay people reading your blog the term ‘bias’ is likely to be interpreted as suggesting a deliberate skewing of results rather than poor procedure. Is there a reason you’ve chosen to use the term ‘bias’ in this article rather than ‘house effect’?
The most important thing to note is that there are two separate terms here — “house effects” and “bias” — that I use in distinct ways.
When I talk about a house effect, I mean what results a pollster shows relative to those of other polling firms. If everyone else shows the Republican Sharron Angle ahead by 3 points in Nevada and some other pollster comes along showing Harry Reid ahead by 5 points, that would be an 8-point Democratic-leaning house effect.
By bias, I mean how a pollster’s results compared with the actual outcome of the election. In the example above — since Mr. Reid won by about 5 points — the pollster showing him winning by that margin would have an unbiased poll, even though it had a strong house effect. In contrast, the pollsters with Ms. Angle ahead would have a strong Republican-leaning bias, even though they hadn’t had any house effect.
Put another way: house effects are what we look at before the election; bias is what we look at after the election.
House effects are undoubtedly predictive of bias to a certain extent, but you’d be surprised by the number of exceptions.
For instance, SurveyUSA actually had a stronger Republican-leaning “house effect” than Rasmussen Reports did in the run-up to the election. But its polls turned out to have rather little bias (less than a full point in the direction of Republicans) once we compared them against the actual results, whereas Rasmussen’s was rather severe (about 4 points).
This is mostly because the two firms were surveying a different mix of races: SurveyUSA did almost half its polling in House races of individual Congressional districts, whereas about 97 percent of Rasmussen Reports polls were of Senate or gubernatorial races. It turned out that while Senate and gubernatorial polling was somewhat biased against Democrats this year, House polls were biased somewhat toward them. So what SurveyUSA was telling us was actually quite valuable — there’s going to be something a little different about how House and Senate races behave this year, it was saying, with Democrats underperforming in the former but overperforming in the latter, which turned out to be the case.
For Rasmussen, however, the Republican lean in its polls ran pretty much wire to wire. It had a significant Republican house effect early in the election cycle and a significant Republican house effect late in the election, and it would up turning into a significant Republican bias on Election Day.
One reason Thursday’s column was pretty unforgiving to Rasmussen is precisely because the Republican lean in its polls was evident so early on. When a particular survey firm shows a persistent house effect, it can basically do one of three things to to exculpate itself:
1. The firm’s pollsters can explain why the house effect exists. That is, why they think they are right and everyone else is wrong — or at least, why they think they’ve created a model of the voter universe that is based on reasonable assumptions.
SurveyUSA, for instance, in explaining the Republican house effect in some of its congressional polls, talked about the presence of what it called “uniquely motivated” voters: Republican-leaning voters who don’t usually vote in midterm elections (and therefore might have been screened out by other companies’ likely-voter models) but were particularly engaged by politics this year (perhaps these were conservatives inspired by the Tea Party).
Similarly, the people conducting the University of Southern California/Los Angeles Times polls in California, which had seemed to have a Democratic-leaning house effect but turned out to be right on the money in projecting Senate and gubernatorial races, explained their belief that the differences resulted from their using bilingual interviewers and doing a better job of capturing the Latino vote.
2. The pollster can re-examine its methodology and correct the house effect. While a pollster probably shouldn’t put its “finger on the scale” and adjust the results of its poll simply so that it seems like less of an outlier, what it can and probably should do is examine its process: whether its difference indeed results from some sort of methodological flaw, or even a bug. SurveyUSA pollsters again provides for a good example here: they were obtaining strange results in their polls in Virginia’s 5th Congressional District, showing the Democrat Tom Perriello down by as much as 26 points (Mr. Perriello eventually lost but by only about 4 points). But by adopting a different methodology — switching from using a registered voter list to a random digit-dialing method — they significantly cut Mr. Perriello’s deficit (perhaps the registered voter list they had been using was out of date or otherwise faulty).
Another example is Gallup, whose traditional likely voter model showed a 15-point Republican advantage on the generic ballot — much higher than most other pollsters. This estimate wound up doing quite poorly: Republicans will win the national House popular vote by about 7 points instead. Gallup, however, was at least willing to publish results with a different likely voter calculation (which showed Democrats down by 10 — not such a bad miss), while also continuing to show results among registered voters, just so you knew where they were coming from.
Although this hedging was slightly annoying from the standpoint of aggregating polls (which version of its likely voter model should we use?), it reflected a willingness on Gallup’s part to admit that it could potentially be wrong.
One of life’s little ironies is that, over the long run, people who are willing to admit they could be wrong turn out to be wrong a lot less often than people who aren’t: the same is true in polling.
3. The pollster can do neither of the above but shut everyone up by being spectacularly right on Election Day. This should be self-explanatory.
Rasmussen Reports didn’t do any of these three things. Its pollsters didn’t provide cogent explanations of why their results were different; the only explanation they offered — that it had to do with their likely voter model — turned out not to hold water. Rasmussen also didn’t “fix” its house effect: it was quite persistent throughout the whole cycle. And their polls did quite poorly, rather than extremely well, on Election Day.
Nor has Rasmussen been any more willing to engage in a discussion about the issues in its polling after the fact. When I asked Scott Rasmussen for comment Friday morning, he wrote me a terse e-mail that said he “can’t imagine any need to respond,” and he has been similarly dismissive with other reporters.
I’d add a fourth dimension here, although it’s not unrelated to the other three. When a particular polling firm does have a strong house effect, I think it’s worth looking at its “fundamentals,” i.e., whether its methodology seems inherently sound. Rasmussen also fails this test: it takes so many shortcuts and violates so many polling conventions, that you wouldn’t really expect its polling to be very good over the long run. Perhaps, if all the little things that traditional pollsters worry about turn out not to matter all that much, Rasmussen’s polls would be roughly as good as others, and they could provide polling much more cost-effectively (this is the gamble that Mr. Rasmussen’s business model is based upon). But you wouldn’t expect its results to be better than everyone else’s. So if its results diverged from everyone else’s, you’d have some reason to suspect that it was was wrong and everyone else was right, and not the other way around.
Richard S., our reader, also raises another question about our use of the term “bias.” Even if we can distinguish “bias” from a “house effect,” isn’t “bias” a bit of a loaded term, especially in a political context?
I appreciate his point. “Bias” when used in casual, everyday sense usually conveys something akin to partisanship or some deliberate departure from objectivity.
But “bias” also has a meaning in a statistical context: it is precisely the term, in fact, that statisticians use to describe a systematic difference between expected and actual results, and there is no particularly good substitute for it. This is the sense in which I use the term.
I don’t come to any judgment about whether statistically biased results happen in any particular instance to reflect partisan bias. Certainly, in Rasmussen’s case, methodology issues would probably be sufficient to explain the presence of a significant statistical bias in its results, irrespective of Mr. Rasmussen’s ideology. There is also an in-between case in which a researcher leaves his model be if it produces results that he “likes” but tinkers with it (re-examines his assumptions and so forth) otherwise. This type of implicit departure from objectivity is probably much harder to avoid than overt partisanship and it is also probably also much more common.