For a long time, I’ve had a very low opinion of Internet-based polling. This is mostly driven by my experience with Zogby’s interactive polling, which has performed abominably at every juncture.
In compiling the data for my new pollster ratings, however, I have discovered that two other Internet-based pollsters — Harris Interactive and YouGov (the latter is sometimes branded as Polimetrix or as the Economist) — are capable of producing decent results.
The chart below documents the performance of the various Internet-based pollsters on occasions when they released a non-trivial amount of polling. It is compared against the average performance of all telephone-based polls during that particular set of elections. In both cases, polls are included in the database if their median field date was within 21 days of the date of the election. Only state-level polls (no national polls) are considered.
Zogby Interactive polls are perhaps two orders of magnitude worse than typical telephone polls. In predicting the final margin between the two candidates, they’ve been off by an average of 7-8 points, whereas 3-5 point errors have been more typical for telephone pollsters. In a world where the difference between “great” and “OK” pollsters might only amount to perhaps an extra half-point of accuracy on average, that’s an enormous discrepancy.
Before Zogby, however, there was Harris Interactive. They first experimented with Internet-based polling in 1998, with decidedly underwhelming results. Their polls showed a significant “house effect” toward Republican candidates, and they were off by an average of 7.1 points in Senate races and 9.7 points in gubernatorial races. With that said, 1998 was a very tough year for most of the traditional pollsters as well, and Harris was hardly alone in having difficulties.
In 2000, Harris came back to conduct surveys in 38 states, capturing the majority of Presidential, Senate and gubernatorial races that were held that year. And in contrast to 1998, they actually did quite well, beating the telephone pollsters by 1.4 points for the Presidential and Senate contests, and 0.6 points in the handful of gubernatorial races. They were in fact among the best-performing pollsters of the cycle.
In spite of this solid performance — which received some notice in the academic literature — Harris pulled back from doing state-level polling. They released polls in just three states in 2004 (not shown in the chart above), missing by an average of 5.2 points. Since then, Harris Interactive has continued to release national surveys with some regularity, but no data at the state level.
I spoke (by e-mail) with Jon Siegel and John Bremmer at Harris Interactive today, who had graciously sent their comprehensive polling from 2000, and asked why they had withdrawn after seeming to have had some success. They explained that it was strictly an economic decision — Harris is a diverse company that had not traditionally focused extensively on U.S.-based electoral polling, and they weren’t persuaded that coming up with state-level estimates — which they say isn’t cheap when done properly — was worth the additional expense.
In the meantime, YouGov — which enjoys a reasonably good reputation in the UK but maintains a lower profile here — has come along. In 2006, both their Senate and gubernatorial polls performed up to the standards of telephone-based polls. Their state-level Presidential polling was marginal in 2008 — they missed by an average of 4.6 points points, versus 3.4 points for a typical telephone poll — but their Senate polling was just fine.
Overall, I think I’ve been too harsh about lumping all Internet-based polls together. Zogby’s method — which allows people to sign up for surveys voluntarily — is on the verge of being unscientific, and is evidently incapable of being worthy of any attention other than as a curiosity. But YouGov and Harris Interactive, which use more careful designs to correct for self-selection bias, are probably not far behind typical telephone-based polls, and may already be ahead of some of the less rigorous ones. And the gap will arguably narrow as Internet penetration becomes more widespread at the same time that telephone polls are having lower and lower response rates and missing increasing swaths of the population.
Frankly, I think the medium-to-long-run future of polling is likely to involve some combination of telephone and Internet polling, perhaps including hybrid polls that incorporate both mediums within the same survey. What the acceptable performances of YouGov and Harris Interactive suggest is that Internet polling is far from hopeless, and over time might come to be quite good. On the other hand, the performance of Zogby suggests that the price may be very great for shoddy or unthoughtful methodology, to the point that such results may be nearly unpublishable.