How closely are Americans following the story of Elena Kagan’s nomination to the Supreme Court? Two different pollsters provide two very different answers.
On the one hand, we have this recent poll of American adults from CBS. According to that poll, just 11 percent of Americans are following Kagan’s nomination very closely; another 35 percent are following it somewhat closely.
On the other hand, we have this survey of likely voters from Rasmussen. In that poll, 37 percent are following Kagan’s nomination very closely — more than three-and-half times the CBS figure:
Of course, these polls are not directly comparable. CBS surveyed all adults, whereas Rasmussen surveyed what it describes as likely voters. Notwithstanding that Scott Rasmussen has described his likely voter model as “fairly loose”, could this account for the differences?
Let’s take the CBS poll as our baseline. Suppose that 100 percent of people in the poll who say they are following Elena Kagan’s nomination “very closely” are likely voters. (This doesn’t seem entirely implausible; it’s a fairly boring story for anyone other than political aficionados.) Suppose furthermore that 70 percent of people who are following the story “somewhat closely” are likely voters, as are 40 percent of those who are following it “not very closely” and 10 percent of the people who are following the story “not at all closely”. (This would imply that about half of the adult population are likely voters overall).
If we apply this likely voter “model”, the CBS poll reconstitutes itself as follows:
This is certainly much closer to the mark. Still, there are some significant differences. Even with these very aggressive-seeming assumptions, we still only get the pecentage of “very closelys” up to 22 percent — not to the 37 percent that Rasmussen shows.
The discrepancy probably arises from self-selection bias. I’ve never received a call from Rasmussen, but from anecdotal accounts, they do indeed identify themselves as being from Rasmussen and I’m sure that the respondent catches on very quickly to the fact that it’s a political poll. I’d assume that someone who is in fact interested in politics are significantly more likely to complete the poll than someone who isn’t.
Of course, that should be a problem with the CBS poll too — and to some extent, I’m certain that it is. But they have some advantages. They use a live interviewer, who will probably elicit more patience from the respondent. They’ll call you back if they miss you the first time around. They’ll use selection procedures to generate a random sample, rather than simply speaking to the person who happens to pick up the phone. And they’ll call throughout the course of the day, not just between 5 and 9, which happens to be the peak time for news programming. They may also be more sophisticated about using weighting procedures to mitigate their response bias, but who knows.
In general, if you’re trying to understand what makes Rasmussen polling “different”, the key heuristic is to assume that their polls are suffering from significant self-selection bias, and that the people who respond to their polls are significantly more likely to be active consumers of political news. This is probably why Rasmussen polls tend to show extremely large “bounces” associated with seemingly banal political events, and why they tend to show good results for candidates associated with activist movements, even if those candidates are barely known among the broader public. In essence, they’re about half-way toward being polls of political junkies. (I’d love to see the percentage of people in their polls who claim, for instance, to have donated to political candidates, something which we could cross-check against FEC records; I’d bet you that it’s very high.)
This isn’t entirely a bad thing. Political junkies are almost certain to vote (which is why it’s redundant to apply a likely voter screen). And they may be “early adopters” for candidates and causes that will penetrate into the mainstream later on. Still, the more response bias a poll suffers from, the more potential there is for it to manifest itself as partisan bias.