In 2015, two grad students at the University of California, Berkeley set out to study the art of political persuasion. It’s not easy to do this kind of research as it happens (or doesn’t happen) in the real world. Most of the time, when scientists study persuasion, they do it in a lab where the situation can be tightly controlled and the variables accounted for. But David Broockman and Joshua Kalla thought they had an easy solution. Little did they know, that simple fix would lead them to set off a scandal and, eventually, help advance political science.
A UCLA grad student named Michael LaCour and Columbia University political science professor Donald Green had just published some game-changing research showing that straight people who met a real, live gay person and listened to that person’s story could be persuaded to support marriage equality, and that this change of heart could be long lasting. The study was a big deal. The news made The New York Times. It was all over the web. It became the basis for an episode of “This American Life.”
Broockman figured he could save time and effort by just using LaCour and Green’s methodology. He even partnered with the same political organization, the Los Angeles LGBT Center. He studied LaCour’s paper like a Talmudic scholar, seeking insights that would help him set up his own experiments. And that was how David Broockman figured out Michael LaCour likely fabricated his results.
“We weren’t getting a great response rate, and we had no idea why,” Broockman told me. “We figured there must be some magic recruitment approach that the survey vendor had used. So we called up uSamp” — the third-party company LaCour said he used to conduct his surveys — “and we say, ‘Hey, whatever you did for him, can you do that for us?’ And they say, ‘Uuuh, we never did that for him.’”
Instead of Broockman being the student, learning from LaCour’s example, he became LaCour’s accuser. In May 2015, Broockman published a 27-page report that included evidence LaCour had failed to survey participants after they were exposed to the persuasion technique and had, instead, simply made up the results. LaCour has not admitted to this. FiveThirtyEight reached out to him for a comment, but he has not yet offered one.
Now, Broockman (who has since graduated and become an assistant professor at Stanford) and Kalla (still a grad student) have finally published the study they’d originally set out to conduct. Their results are striking, both for their extreme transparency and in the way that they do (and don’t) support LaCour’s imaginary data. It turns out LaCour was right that minds can be changed, but he was wrong about how. What’s more, this experience led Broockman to come up with a new methodology that has far-reaching implications for the whole discipline of political science field research — one that is easily replicable and could cut the cost of these kinds of studies by tens of thousands of dollars.
Like LaCour and Green, Broockman and Kalla were specifically studying the effectiveness of a political canvassing technique developed by the L.A. LGBT Center. Both Broockman and David Fleischer, director of the center’s Leadership Lab program, said that canvassing is usually a quick and dirty enterprise. Canvassers show up at a door, they give a speech, they try and accomplish a goal, and they get through it all as quickly as possible. “There’s a reason why when people knock on your door you aren’t immediately thrilled,” Fleischer told me.
The center has a different technique, one that’s structured more like a Socratic dialogue and can take as long as 20 minutes to get through and on average lasts 10 minutes. Canvassers are aiming for a conversation, in which they ask questions and the subject gets to talk. They don’t tell people ahead of time what conclusion they want to reach. There’s no sermon built in. The goal is that, by the end, subjects will have built up empathy with a group of people different from themselves.
In the case of this new study, that group was transgender people. Canvassers built empathy by leading cisgender people1 — those whose assigned-at-birth gender matches the gender they understand themselves to be — to think of times when they were judged by others and, then, to connect those feelings to how transgender people feel when they are judged.
Broockman and Kalla used a unique methodology to test the efficacy of this technique — one that Broockman says made the testing drastically cheaper than usual. Only about 20 studies over the last decade have used surveys to understand the efficacy of persuasion techniques in the real world. According to Broockman, each of these studies had methodological shortcomings that not only resulted in flawed data but also made each cost upwards of $2 million to conduct.
These past surveys typically chose a random group of voters and then split them into a treatment group and a control group. But it’s really common for people to turn canvassers down. Because of that, researchers end up having to try to canvass huge numbers of people, just to get a sample size big enough to be useful. Meanwhile, the follow-up surveys allowed scientists to see how respondents’ thinking changed after the canvasser left, but they had no way to know how it compared to their beliefs before the doorbell rang.
Broockman solved those problems by implementing a pre-selection process. He sent out mailers, inviting people to participate in a broad online survey. Only a couple of questions had anything to do with transgender rights issues. But, by taking the survey, people signaled to Broockman that they were more likely than the general population to open the door for a canvasser later. That meant fewer people had to be canvassed to reach the same power of result. The downside to this: Broockman doesn’t know whether his sample is biased in some way. When I suggested that people who are interested in taking surveys might be more interested in questioning their own beliefs, he agreed that was a risk. On the other hand, that would also be true of the control group, not just the treatment group, and now Broockman knew something about those people’s baseline attitudes about the transgender community.
“If you wanted to do a study like this five years ago, it was going to be $2 million. The way Michael [LaCour] supposedly did it cost $1 million. Now, we realized that we could do it for more like $25,000,” Broockman told me.
Our podcast What’s The Point discusses the latest developments in the LaCour scandal (begins at 6:00).
The study used a common political science tool called a “feeling thermometer” to quantify emotions. Three days later, people who had experienced the L.A. LGBT Center’s persuasion technique showed an average 10-point increase relative to the control group in their positive feelings about transgender people, on a scale of 100. Three months later, that average 10-point increase in positive feelings persisted. For comparison, between 1998 and 2012, Americans’ positive feelings about gay and lesbian people increased by an average of 8.5 points. That change came about slowly, through a combination of cultural influence, explicit attempts at persuasion and implicit peer pressure. It’s considered to be one of the biggest success stories in the history of political persuasion, said Diana Mutz, director of the Institute for the Study of Citizens and Politics at the University of Pennsylvania. “It’s impressive that [Broockman and Kalla] were able to change minds in a short period of time and that it persisted,” she said.
Of course, everybody said the same thing about the LaCour and Green paper. Unlike LaCour, though, Broockman and Kalla have published their research in an exceptionally transparent way — making his group’s data and code publicly available for anyone to see. The result is a “landmark study of scientific transparency,” said Elizabeth Levy Paluck, a psychologist at Princeton who wrote a commentary that accompanied the new study in Science. “They really dotted all their i’s and crossed all their t’s.”
The other difference between LaCour’s paper and Broockman’s is more subtle. LaCour found that the L.A. LGBT Center’s technique only worked when the canvasser using it was gay or lesbian. LaCour hypothesized that this was an example of the power of intergroup contact, an idea that Broockman summarized thusly: “David Broockman, a gay man, can go talk to you and you say, ‘David is so charming! Now all gay people are great!’” LaCour even did a follow-up study on attitudes toward abortion that found the same thing: Canvassers who had had an abortion, and disclosed that fact, could change people’s minds.
But that is not what Broockman found at all. For one thing, his study’s big changes in attitude about transgender people were produced by canvassers who were mostly cisgender. There wasn’t any difference in the results produced by cis versus trans canvassers. And Broockman also did a follow-up to see if the same canvassing technique could be used to change peoples’ views on abortion, but this time the technique completely failed.
Nobody knows why this method worked for transgender issues but not for abortion, but Broockman and Fleischer suspect it had something to do with perspective-taking, their alternative explanation to LaCour’s “intergroup contact” idea. Broockman and Fleischer think the new technique’s success was dependent on getting respondents to make themselves vulnerable by sharing an experience of their own, and then leading them to imagine themselves in a trans person’s shoes. The abortion study failed, they believe, because they just haven’t had as much practice at getting people to that place on that issue. It’s about the skill of the canvasser, not the identity of the person doing the canvassing, Broockman told me.
But Mutz disagrees. Perspective taking isn’t something new that the LGBT Center invented, she said, just as LaCour didn’t invent intergroup contact. “There’s lots of evidence that both are effective,” she told me.
Instead, Mutz thought the center’s technique was unlikely to ever get big results on abortion because that issue was much more entrenched, much more divisive and much more of a bellwether for identity politics than trans rights are at this point. It’s always harder to change an opinion when someone has held it for a long time. It’s possible, she said, that in a few years it won’t be as easy to change people’s minds about transgender rights.
At this point, it’s hard to say what’s going on. That’s why Broockman told me that the methodology he developed to do the study was almost more important than the results of the study, itself. “Right now, we have just two data points,” he said. “The problem with that is that any narrative you want can fit with them. But the method will allow for everybody to have their version of this study.”
Ironically, he wouldn’t have developed that methodology without LaCour’s work. Broockman even attributes the idea of a precanvassing survey to LaCour and Green. Altogether, it’s a model example of how the process of replication advances science, even when that process uncovers flaws or even fraud. Now other research teams get a chance to test the methods again and take the science a step further. None of these pieces individually will give us absolute truth, but taken together, they’ll get us closer.
CORRECTION (April 7, 2:44 p.m.): An earlier version of this article misidentified David Broockman’s position at Stanford. He is an assistant professor, not an associate professor.