A $4.4 million Pentagon survey which asks its troops a series of questions about Don’t Ask Don’t Tell (DADT) has come in for a lot of criticism, mostly related to purported bias in the selection and wording of questions. I might weigh in at more length on the bias claims later; my initial position, having read the survey, is somewhat agnostic. But really, I think the bias accusations somewhat miss the forest for the trees. The survey might or might not be biased — the bigger problem is that entire parts of it are completely useless.
In particular, given that the army that isn’t supposed to have any (openly) gay soldiers, the survey asks the troops to engage in an awful lot of speculation about the gay soldiers in their midst. Typical are questions like these:
Do you currently serve with a male or female Service member you believe to be homosexual.
There are several different version of this question — the respondent is asked about leaders, coworkers, and subordinates whom he deems to be homosexual, and in turn asks what impact these purported homosexuals are having on his unit’s performance and morale.
It’s good, I suppose, that the survey seems to acknowledge that there are in fact gay people currently serving in the military (statistically speaking, there must be thousands of them). But what it doesn’t really ask is how the respondent has come to his knowledge of this. In practice, there are a range of circumstances that might lead one to conclude that there is a gay person in his unit, running the gamut from direct evidence, to indirect evidence, to hearsay evidence, to speculation based on common stereotypes:
You might think Solider Q is homosexual if…
[Definitive evidence]
— You’ve had sex with Soldier Q.
— Soldier Q has told you that he’s gay.
— Soldier Q identifies as gay on his facebook or MySpace profile.
— You find gay pornography that seems to belong to Soldier Q.
— You find what seems be a love letter from a same-sex partner in Soldier Q’s possession.
— You’re pretty sure that Soldier Q made a pass at you.
— Soldier P claims that Soldier Q made a pass at him.
— Soldier P claims that he saw Soldier Q walking into a gay bar on shore leave.
— Soldier P tells you that “everyone knows” that Soldier Q is gay.
— Soldier Q has a number of close gay friends.
— Soldier Q listens to music that is popular with gay people.
— Soldier Q likes to tell homoerotic jokes.
— Soldier Q has some pretty effeminate mannerisms.
— Soldier Q just seems a little bit peculiar.
[Dubious evidence]
Undoubtedly, the circumstances toward the top of the list do occur occasionally. Somewhere in the world, two male or two female American soldiers are having sex with one another, or have confided their sexual feelings to one another. And surely there are other soldiers, particularly in reserve or noncombat units, whose sexual orientation is an “open secret”. But considering that the penalty for either homosexual conduct or open disclosure of one’s homosexuality is expulsion from the army, one would assume (i) this occurs somewhat rarely; (ii) when it does occur, the troops take every precaution to keep it a secret; (iii) these precautions might extend to discretion/dishonesty on a survey that offers somewhat dubious guarantees of privacy and anonymity.
On the other hand, there’s nothing to prevent a solider from speculating that some of his colleagues are gay, and this must occur rather frequently. Indeed, the survey almost seems to encourage troops to use their “gaydar” — the question it poses is not whether they know other troops to be homosexual, but whether they believe them to be homosexual. Occasionally, there might be some relatively decent evidence for this, or there might be some dubious evidence that just so happens to be right. But for the most part, it would seem that you’ll be picking up a tremendous number of false positives — soldiers who are believed to be gay, but aren’t — and that these false positives will swamp any instances in which soldiers (in spite of DADT) are actually somewhat open about their same-sex attractions.
What further seems likely is that the distribution of false positives will be biased. In particular, it seems likely to be biased in two ways:
— Soldiers in units with low morale are probably more likely to accuse one another of being gay based on hearsay evidence, i.e. rumors or innuendo. Although this might be changing some, there’s usually no better way for one young man to undermine another than to allege that he is gay. And these (usually spurious and false) accusations would presumably be more common in units where the troops weren’t getting along with one another. Thus, the survey will probably attribute lower morale to the presence of homosexuality within the unit, when in fact it’s low morale that triggers the suspicion (but not the actual fact) of homosexuality.
— Homophobic soldiers are probably more likely to accuse one another of being gay based on stereotypes. Our troops range from relatively worldly young men and women to others who are teenagers and who have barely seen the world outside of their hometowns. They might mistake behaviors (one’s choice of music, for instance) that in fact reflect socioeconomic or cultural differences for instead being indicators of sexual orientation. Or their homophobia might be less benign: they see gays lurking around every corner. If they don’t get along with their commander, for instance, they might put one or two facts together (he isn’t married and likes watching Sex and the City!) and conclude that must be gay.
Thus, this entire portion of the questionnaire is fairly useless: it more measures the relationship between gossip and unit morale than anything having to do with homosexuality per se. Although this might be modestly interesting as a sociological experiment, any conclusions that it purports to come to about the impact of DADT on morale should be ignored.
In contrast, the survey (at least from what we’ve seen of it so far) goes out of its way to avoid asking the troops about something which is arguably more relevant and which is certainly more measurable: their opinions about DADT. At no point, for instance does it pose the simple question of whether or not the solider thinks that DADT should be repealed. I’d have no huge problem if we asked our troops that; it would be up to our policymakers to weigh those findings against other factors. But the survey does not solicit the soldiers’ opinions; instead, it solicits their speculation on the sexual preferences of their peers. In so doing, it insults their intelligence — and ours.
EDIT: A few people down in the comments who are accusing me of missing the point are in fact missing the point themselves — although perhaps my writing was unclear. Arguably it would be useful to determine how soldiers would react in the event that there were openly gay people in the army. But the reaction of soldiers who think they’re serving with a gay person now is an inappropriate proxy for this, for two reasons. One, because the sample of soldiers who think they’re serving with a gay person now (in most cases, they won’t really know) is almost certain to be biased, and might disproportionately include those troops who are (a) homophobic, or who (b) serve in a unit with low morale. And second, because even if the soldiers somehow had perfect gaydar and knew exactly who was gay and who wasn’t, the effects of gays serving openly might be very different than their serving while in the closet (as almost all of them will be doing in the status quo).
It’s a bit like trying to anticipate a community’s attitude toward a potential influx of Russian immigrants by asking them whether there are any Russian spies around, and if so, what impact they’re having on the neighborhood. If you did that, the people who said there were Russian spies would probably be disproportionately Russophobic (and/or simply paranoid); their attitudes toward Russians would not be remotely representative of the community as a whole.