A pancreatic cancer diagnosis is basically a death sentence. The five-year survival rate is 6 percent. A prostate cancer diagnosis, on the other hand, offers patients more room for hope — its five-year survival rate is nearly 100 percent. But despite the difference in the severity and prevalence of these diseases, the approval process for drugs to treat them, and any other disease, is the same.
New research from a pair of MIT economists argues that the Food and Drug Administration, which decides which new drugs get to market through randomized clinical trials, is too one-size-fits-all in its approach to the approval process for treatments for everything from leukemia to the flu. As a result, they say, the agency is too conservative in regulating trials for severe diseases like pancreatic cancer and not conservative enough when it comes to drugs for less dire ones like prostate cancer.
“The FDA is staffed by extraordinary individuals, but we’re focused on providing them with another tool to focus on desperate patients who have no other recourse,” said Andrew Lo, a professor of finance at MIT and the author of the new working paper. Lo, along with his student Vahid Montazerhodjat, argues that the agency’s judging of clinical trial requirements by a uniform statistical standard is a mistake. Instead, they say, the standards should take into account the severity of the diseases that the drugs seek to address. The Washington Post had a nice write-up of the paper’s implications, and Lo’s statistical approach to the problem is worth exploring further.
It might seem strange that Lo, known for his research on hedge funds, is now studying drug trials. But he gave a personal reason in an interview this week: “My motivation for doing this research was seeing friends and family members die of cancer and them not having access to experimental drugs.”
The FDA must strike a difficult balance between approving effective drugs quickly and not exposing patients to drugs with bad side effects. In striving to meet these potentially conflicting goals, the FDA can make errors of two types: approving ineffective or harmful drugs and rejecting effective ones. The first type of error is a “false positive” (or Type I error, in statistics jargon); the second kind is a “false negative” (or Type II error). The standard threshold for approval in an FDA clinical trial is a false positive rate of 2.5 percent or less; in other words, the FDA is willing to tolerate approving ineffective drugs 2.5 percent of the time.
But Lo and Montazerhodjat think the threshold for a successful trial should vary from disease to disease. For severe diseases, the researchers think the false positive rate should be higher, to allow more drugs to hit the market even though some of them would be ineffective or harmful. A patient with a seriously life-threatening disease like lung cancer is perhaps more willing to gamble on a risky drug in pursuit of a cure, while someone with a disease that has a high survival rate such as diabetes presumably cares more about avoiding adverse side effects.
Lo and Montazerhodjat used data from the 2010 U.S. Burden of Disease Study to estimate false positive rates for different diseases depending on their severity. The chart below shows their recommended thresholds for a number of diseases. The severity score is on a scale of 0 to 1, with 0 representing “no loss of health” and 1 being death; prevalence is the number of people afflicted in the U.S. Because pancreatic cancer, for example, is such a terrible disease, their method argues that drugs should be approved from trials risking a false positive rate of 27.9 percent or less — a far more aggressive target than the FDA’s 2.5 percent. “Imagine if I had pancreatic cancer,” Lo said. “I’m willing to take a 1 in 4 chance the drug you give me is not going to work. Because the alternative is: I’m dead.”
On the other hand, the FDA’s 2.5 percent threshold is too high, according to this metric, for trials of drugs that treat less severe diseases. Take prostate cancer: Lo’s method says that the FDA’s standard leads to the approval of too many ineffective drugs for treating it and that a false positive rate of 1.2 percent should be used instead.
To create these estimates, Lo and Montazerhodjat used a technique called Bayesian decision analysis, which “tries to weigh the two different types of mistakes one can make and to minimize a weighted average of the two,” Lo said. “There’s nothing new about this approach — we didn’t invent it.”
Donald Berry, a professor of biostatistics at the University of Texas’s MD Anderson Cancer Center, arguably did. Lo calls him “the godfather” of Bayesian approaches to clinical trial design and likes this Berry quote: “We should be focused on patient values, not just p-values,” a jab at the traditional (or frequentist) approach to clinical trials. I asked Berry about Lo’s research, and he agreed that the FDA was too “anal about controlling the Type I error rate.” But, he added, “I’m actually positive about the FDA’s attitude toward these things, in particular for cancer.”
For its part, the FDA resists the allegation that it’s too rigid. The agency declined to comment for this article but pointed to remarks that then-FDA Commissioner Margaret Hamburg made in March about how “clinical trial requirements have been steadily increasing in flexibility.” As evidence, Hamburg cited data on the share of drug trials using “expedited approval processes” last year — 66 percent, the highest level ever. And the FDA has made some progress toward incorporating Bayesian methods, recently issuing guidelines on “adaptive designs” for trials of medical devices.
But a faster approval process is different from what Montazerhodjat and Lo are proposing. Their approach would fundamentally change the parameters of a successful trial.
Statistics aside, the debate over the FDA’s clinical trial design might come down to philosophy. The trade-offs between Type I and Type II errors may seem like dry statistical quibbles, but they present, as the new working paper puts it, “utilitarian conundrums.” If the FDA were more tolerant of false positives in drug trials for terminal diseases, the U.S. might inch toward the greatest good for the greatest number.
“We make life and death decisions all the time,” Lo said, citing speed limits on highways as one example of how the government makes trade-offs between competing aims, in this case travel convenience and safety. For drug trials, the calculation involves potential cures to terminal illnesses. “We make these trade-offs all the time; we just need to do it in a rational way.”