“There’s a difference between an answer and a result. But all the incentives are pointing toward telling you that as soon as you get a result, you stop.”
If you Google “is science broken?” you gather a surprising number of hits. It’s a question born of a perceived moment of crisis: Retractions are on the rise, the gap between what the public believes and what scientists believe is vast, and (perhaps relatedly) the number of scientists who say “this is a good time for science” continues to drop. The media doesn’t help. Headlines tend to overstate findings in search of catchy takeaways — drink eight glasses of water! — when the research is often much more nuanced.
On this week’s episode of our podcast What’s The Point, FiveThirtyEight’s Christie Aschwanden surveys the state of science and what it says about the impact research can have on our daily lives. She recently wrote an article addressing the question of whether science is broken, which also features an interactive graphic by Ritchie King on a particularly troubling technique of data manipulation known as “p-hacking.” Aschwanden concludes that science isn’t necessarily broken, but that it’s very, very hard — the way it always has been.
Plus, this week’s Significant Digit: In 2014, 218 newborns were named Anakin.
Stream or download the full episode above. Below is an excerpt from Christie’s story. And here’s the interview with Neil deGrasse Tyson referenced in the episode.
Science’s moment of crisis
Excerpt from Christie Aschwanden’s article for FiveThirtyEight. Read the full story.
After the deluge of retractions, the stories of fraudsters, the false positives, and the high-profile failures to replicate landmark studies, some people have begun to ask: “Is science broken?”
I’ve spent many months asking dozens of scientists this question, and the answer I’ve found is a resounding no. Science isn’t broken, nor is it untrustworthy. It’s just more difficult than most of us realize. We can apply more scrutiny to study designs and require more careful statistics and analytic methods, but that’s only a partial solution. To make science more reliable, we need to adjust our expectations of it.
Science is not a magic wand that turns everything it touches to truth. Instead, “science operates as a procedure of uncertainty reduction,” said [Brian] Nosek, of the Center for Open Science. “The goal is to get less wrong over time.” This concept is fundamental — whatever we know now is only our best approximation of the truth. We can never presume to have everything right.
“By default, we’re biased to try and find extreme results,” [John] Ioannidis, [a] Stanford meta-science researcher, told me. People want to prove something, and a negative result doesn’t satisfy that craving. Ioannidis’s seminal study is just one that has identified ways that scientists consciously or unconsciously tip the scales in favor of the result they’re seeking, but the methodological flaws that he and other researchers have identified explain only how researchers arrive at false results. To get to the bottom of the problem, we have to understand why we’re so prone to holding on to wrong ideas. And that requires examining something more fundamental: the biased ways that the human mind forms beliefs.
If you’re a fan of What’s The Point, subscribe on Apple Podcasts, and please leave a rating/review — that helps spread the word to other listeners. And be sure to check out our sports show Hot Takedown as well. Have something to say about this episode, or have an idea for a future show? Get in touch by email, on Twitter, or in the comments.
What’s The Point’s music was composed by Hrishikesh Hirway, host of the “Song Exploder” podcast. Download our theme music.