I was troubled by a survey released Monday by Mitchell Research in Michigan. It wasn’t that the results — which showed Democratic Rep. Gary Peters leading Republican Terri Lynn Land by 2 percentage points in the U.S. Senate race and Republican Gov. Rick Snyder leading Democrat Mark Schauer by 5 points in the governor’s race — were necessarily wrong. It was how Mitchell said it arrived at them that bothered me.
From the Mitchell release:
Mitchell Research had intended to release a survey today that we conducted on Wednesday, Sept. 10th, prior to President Obama’s speech to the nation regarding the conflict in the Mid-East. That poll showed Snyder leading by only 1 point, and Peters up by 8 points. However, because of changing poll data nationally, we decided to conduct a survey last night (September 14) to see if those events coupled with the increased television advertising by Snyder and Land might have changed the races in Michigan.
Here’s one way to read this: Mitchell Research conducted a poll, thought the results looked wrong and decided to conduct another survey to get results it thought made more sense.
That would be fine if Mitchell released the full data from the first poll. But it didn’t.
A pollster should release its work regardless of whether it thinks the results are right. Outliers happen even to the best pollsters. They are supposed to happen. And, of course, a pollster has no way of knowing whether a result is an outlier. Sometimes when a poll appears to be an outlier, it’s the first survey to pick up real movement in a race. And sometimes when a poll fits neatly into previous surveys, all the surveys end up being wrong.
Squashing results, however, suggests a pollster is looking at other pollsters’ work (beyond its weighting mechanism) to determine whether data is worthy of publication. Such decisions can lead us down a bad path.
When a pollster holds back some data, we can’t be sure what other results it might be holding back or changing. What might have happened if Mitchell Research’s second poll didn’t seem to match what it thought it should be? Would it have conducted a third poll? What happens when there are other results Mitchell Research doesn’t like?
And though it’s beside the point, I should note that Mitchell’s explanation for the change in results has no backing from the majority of other polls. President Obama’s approval rating is the same as it was a week ago, according to the aggregate of polling information. There hasn’t been a big swing to Republicans in the majority of other Senate races. The FiveThirtyEight estimate actually gives Democrats a better shot at holding on to the Senate than they had a week ago.
But not releasing all results that were meant for public consumption is sketchy, even if it’s thought that those results might be wrong.