Veronique de Rugy has issued a fairly gracious response to my critique of her study on the disbursement of stimulus funds, the crux of which was that she had failed to account for a variable (the presence of a state capital) that was extremely important in predicting the allocation of stimulus funds (because much of the money is intermediated by state governments).
Most importantly, she has promised to evaluate some of my concerns and to re-run her analysis. This is terrific — and she is to be commended for her responsiveness. de Rugy is also to be commended for having released portions of her dataset** on the Mercauts Center website (something which she had done originally). Nevertheless, some further comment on her response — and the issues in research design that her study raised — is warranted:
— I share de Rugy’s disappointment with the quality of the data available at recovery.gov. Frankly, I am not sure that testing her hypothesis to a peer-reviewable level of robustness is possible given the middling quality of data and the inherent ambiguity with how particular projects must be assigned to particular congressional districts.
— de Rugy writes: “The unemployment data for the regressions has in fact been used by congressional district, not by MSA. The confusion comes from the fact that the Excel file on the website includes unemployment by MSA.” Good: that particular issue is cleared up, as well as the reason for my confusion.
— For me, personally, the notion that the allocation of stimulus funds could have reflected a broad-based and widespread effort to benefit districts represented by Democrats seems implausible — something which is well worth examining but something which should have received especially rigorous scrutiny. This is particularly so given that many of the funds were intermediated by state governments, not all of which are controlled by Democrats, as well as federal agencies that were constrained by formula rules.
There are two other variations that I find less impluasible:
I find it less impausible that the funds could have been directed toward those sorts of districts which tend to vote Democratic (e.g. as measured by PVI or by Obama vote share) — even after controlling for other demographic variabes — a possibility that de Rugy raises in her response but which was not the focus of her hypothesis. The difference is that that this could have resulted from a sort of unconscious bias in the design of the stimulus rather than a deliberate conspiracy.
I also find it less implausible that some *particular* projects could have been directed toward those districts that had a Democratic representative who was either especially influential or who a key swing vote in the House. (This is what we call pork.) However, de Rugy ran various tests on the types of Democratic districts that benefited from the stimulus and did not find any relationships with the characteristics of the Democratic members of Congress that tended to represent them.
— There are a few passages in the response where de Rugy is still taking her initial results a bit too credulously, such as:
“So even after I use his methodology I will find that Democratic districts, other than state capital ones, are getting 30 percent more than Republican ones. That does seem like a possible political bias to me, which would be worth looking into.
How much of a bias? I don’t know. Let’s not forget that my take on the data has always been the following: The regression analysis shows that district’s party representation matters. However, I cannot say how much it matters compared to other factors (such as the formula used by different agencies).
Her results suggest correlation — but until confounding variables are rigorously tested and rooted out, they are no proof of casuation at *any* order of magnitude. (It is emphatically not just a question of how much bias there was.)
— de Rugy seems more concerned than I am about my four word-aside (“and possibly deliberately biased”) that raises the question of whether her research design could have been the result of any deliberate political bias. She says it wasn’t and I take her at her word, particularly given her fairness and transparency in responding to me. But I don’t really see raising the possibility that the bias was deliberate as being particularly “inflammatory” — it was manifestly a *possibility*, given how obvious the design flaw was relative to how smart and capable de Rugy obviously is.
Far more likely, however, is unconscious bias: how hard do you push back and cross-check your assumptions when you initially come up with a research finding that you “like” (or one that you don’t like)? This kind of bias, almost by definition, is very hard to avoid, and potentially threatens the work of virtually every social scientist, not just de Rugy.
— de Rugy is correct that many demographic variables are correlated with one another, which makes model speification more difficut and can lead to potential problems with overfitting. However, these demographic variables are also correlated with the poltical representation in the Congress. Moreover, because the stimulus consists of many different ‘layers’ (categories of projects), it is quite plausible that many different demographic variables (as well as intercations between two or more such variabes) could come to bear on how funds were ultimately distributed.
This is a sticky (albeit common) problem. The best way to handle it would probably be to make several different specificiations of the model and to publish them explicitly. If you have five different model specifications, all of which have roughly the same explanatory power, publishing only the one that you most like can potentially reflect bias. I have no idea how many other model specifications de Rugy tested and how many of them might have relieved the partisanship variable of its significance: it is not uncommon for a variable to go from *highly* significant to *completely* insignificant when a new variable with which it is correlated is introduced.
— In general, there are a lot of things that de Rugy could have done — both in terms of her research design and in terms of her presentation — to give one more confidence that she had rigorously cross-checked and scrutinized her design, assumptions, and findings. If this were a chickenscratch, reasonably well-caveated, back-of-the-envelope blog post, I would have been more gentle in my critique. However, given that she has used her study to testify before the Congress, I believe it proper to hold her to a fairly high standard. That she is now willing to revisit her assumptions speaks highly of her, but until she does, her original study should be given no deference.
** Although its a relatively minor point, she did not, in fact, release the entirety of her dataset — such as the economic variables she used in her regressions.