In light of the New York State Senate’s decision on Wednesday to vote down gay marriage, I’ve been thinking a little bit more about what happened last month in Maine, where voters passed Question 1 — reversing the state legislature’s decision to allow for gay marriage — by roughly a 5-point margin.
We had previously developed a statistical model, based on the result of ballot initiatives in other states, which predicted that gay marriage would most likely be upheld in Maine. The model was somewhat well hedged, giving itself as much as a 1 in 3 chance of being wrong depending on what assumptions were made. Nevertheless, this was certainly a disappointing result — from both a policy and a forecasting perspective — especially considering that the vote wasn’t actually all that close. It’s one that’s worth a little self-reflection.
There are essentially three conclusions that we might come to in evaluating the model’s performance:
1) It was a basically good model that got a little “unlucky”;
2) It was a poorly specified model that missed important factors in play in Maine;
3) It was a decent enough model, but missed some sort of national backlash against gay marriage.
The third conclusion is tempting in light of what happened in New York. But there’s not a lot of evidence for it. Back in March, the last time we did a systematic review of gay marriage polling, we found an average of about 41 percent of Americans in support of full marriage rights. Since then, there have been nine polls on gay marriage, which have shown, respectively, 49, 46, 45, 44, 41, 40, 39, 38, and 35 percent support for gay marriage. You see a pattern there? I don’t. It works out to an average of 42, exactly in line with our previous estimate. (Nor is there any particular trendline in the data: the two most recent polls, from NBC/WSJ and AP-GfK, showed gay marriage with 41 and 46 percent support respectively.)
In addition, on the same day that Maine voted down gay marriage, voters in Washington State approved Referendum 71, which greatly expanded domestic partnership rights. And as for New York, we should remember that this was a decision of the state legislature rather than anticipated many months ago.
So maybe the model was just “unlucky” — we did hedge our bets, after all. This conclusion, as always, is tempting. I was rather fond of the model: it was simple and intuitive and yet had a lot of explanatory power. Those are the sorts of models I tend to trust the most (whether I develop them myself or someone else does).
But it has since occurred to me that there was one important factor that the model may have missed — something I’d mentioned in passing but not thought seriously about. In Maine, in contrast to every other state that voted on gay marriage, there was not a constitutional question at stake. Question 1 did not amend the state’s constitution; it merely ‘vetoed’ the law that the legislature had passed. Maine’s Supreme Judicial Court could rule tomorrow that gay marriage was protected by the state’s constitution and gay marriage would again become legal there. Although I’m not certain about this, it also appears that the legislature could again approve gay marriage whenever it wanted. In 1998, for example, a law to ban discrimination on the basis of sexual orientation was overturned by people’s veto in Maine. But a nearly-identical law was passed in 2005 — it went through the people’s veto process again, but this time was upheld.
Does this make a difference? Because Question 1 was the first gay marriage ballot initiative not to impact its state’s constitution, there is no good comparable at the state level. But some national numbers suggest that it the wording of the initiative might have mattered.
There were five polls conducted in 2008 that asked about a federal Constitutional amendment to ban same-sex marriage (nobody has asked this question so far in 2009, so these are the most recent results). And those polls showed that the Constitutional amendment idea was not so popular, with an average of 40 percent of voters in favor versus 54 percent opposed:
Now, compare these numbers to those that we described earlier. Whereas a 54 percent majority opposes an amendment to ban gay marriage (versus 40 percent in favor), a 53 percent majority — between the last nine national polls — also oppose allowing gay marriage (versus 42 percent in favor).
What that means is that there’s a “swing vote” of about 10 percent of the electorate that is not yet ready to allow gay marriage, but is also not willing to ban it (at least not Constitutionally). This is enough to tip the national balance on the question of gay marriage. And it may have tipped the balance in Maine. Question 1 did not actually ban gay marriage but rather overturned the legislature’s affirmation of it.
I realize that this will seem like a semantic distinction, but such distinctions are frequently important in polling and it stands to reason that they might be important for ballot initiatives as well. Besides, the distinction is not entirely semantic; since constitutional amendments have a higher bar to clear in Maine, requiring the approval of 2/3 of the legislature in addition to a majority of the people.
Here, in any event, is what I think we can and cannot say in light of the recent results.
Firstly, I don’t think there is any evidence of a national backlash against gay marriage. It should be borne in mind that gay marriage is still opposed by a small majority or large plurality of Americans. But there’s not really any evidence that the numbers are getting worse; instead, they appear to be v-e-r-y s-l-o-w-l-y getting better.
Secondly, it’s not clear that the results in Maine are comparable to those in other states. Question 1 was the only gay marriage ballot initiative that did not seek to rewrite its state’s constitution. If Question 1 had addressed the same question as Proposition 8 in California, which did alter its state’s constitution, it might not have passed.
Thirdly, building statistical models is tricky. It is good practice to model one’s uncertainty whenever we are forecasting a result. But very often, there is uncertainty about the uncertainty.
In Maine, we had a “known unknown” — I knew that the nature of the ballot initiative was different as compared with other states. But there was no particularly good way to model the uncertainty. And so I did what most people do and ignored it.
Ironically, known unknowns are often tougher to deal with than unknown unknowns. We know we can’t know everything. But because this type of uncertainty is inevitable, we have a host of statistical techniques to handle it; it is essentially what is accounted for by the standard error of a forecast. But sometimes, we know there’s an additional factor or two in play — something that contributes to greater-than-average uncertainty — but we just don’t know how to quantify it. There is not much we can do in these cases but be prepared to be wrong.