Orphan Drug Development: Could We Be Seeing The End Of P Values?
Executive Summary
Panel of experts stresses need to move toward innovative approaches for rare disease drug development such as Bayesian methods, but this would require a paradigm shift in US FDA's regulatory structure.
Could the scientific community be nearing the end of the end of the P value as an efficacy marker in the rare disease drug development space?
It is certainly possible, a panel of experts says. But the key to shifting away from P values toward Bayesian methods would require fitting the approach into the US FDA's regulatory structure.
Speaking at a Duke Margolis Center for Health Policy event March 15 on innovative statistical methods and trial designs for rare disease drug development, a series of FDA officials, industry speakers and researchers spoke of the benefits Bayesian statistics can bring in the orphan arena, where there are often not enough patients for a fully powered randomized clinical trial.
Bayesian statistics involve learning from evidence as it accumulates by combining prior information with current study information on an endpoint of interest, and help to make decisions about the future.
"Pictorially, we can think about this as what we knew, which is out prior; what we see now, that's the likelihood; and then we update that with what we now know," said
Eli Lilly & Co. Research Advisor Karen Price.
Now, Price contends, what the scientific community needs is a formal mechanism to synthesize known information into the decision-making process, and Bayesian statistics are the platform to do that.
"Every time we see results from a clinical trial, we naturally think about other data that we have seen," Price said.
"Whatever it is, we bring that to our decision-making process," she added. "But we do that in our heads, and we determine on our own how much we weight that piece of information."
Lisa LaVange, a professor at UNC Chapel Hill's Gillings School of Global Public Health’s Department of Biostatistics and formerly director of the Office of Biostatistics in the Center for Drug Evaluation and Research (CDER), offered a similar assessment.
She noted that Bayesian methods would allow FDA to formalize using the information it already knows about a treatment, as orphan drug developers don't have the luxury of starting with a clean slate in Phase III trials. FDA currently uses priors in regulatory decision-making in "a very informal way," LaVange says, such as factoring in clinical data from adults into conducting pediatric trials.
"[Bayesian methods are] a way to sort of put your money where your mouth is," Lavange says. "What do I know about this drug? What do I believe about how it will work in this population before I ever start the study.
"It's almost like a cost/benefit analysis. What is the cost of me not using this prior information? Maybe missing a chance to answer the question that I most need to answer."
University of South Florida Professor Roy Tamura argued that Bayesian methods are the only way forward in the rare disease space if the paradigm allows for the inclusion of known information.
"If we are going to allow prior information, external information into the inference, the only way to go is the Bayes way," said Tamura, also a former statistician for Lilly. "And therefore, there are no more P values. We have to move away from that in this space."
What Is Stopping The Shift?
However, LaVange noted, Bayesian methods are still a new exercise for FDA, especially for CDER. For instance, drugmakers and the agency would need to reach an agreement ono a prior probability distribution before conducting a trial.
"So can we have a paradigm shift and have a decision made based on the probability that the drug works given our prior beliefs, our observed data in this trial and the computation of the posterior probability, instead of making a decision that, assuming the drug doesn’t work, what's the chance of me thinking that it works, incorrectly?" LaVange asked rhetorically.
"But we don’t have a tradition of that," LaVange said. "So how high does that probability have to be? 95%? 97.5%? 80%? 60%? Well, maybe it depends on the need and the other drugs that are approved."
John Scott, deputy director of the Division of Biostatistics in the Center for Biologics Evaluation and Research (CBER), also outlined the difficulties about incorporating prior data, specifically with regard to how much information to borrow.
"How do you go about approaching the question of how many patients worth of data should you borrow into the new study?" Scott asked rhetorically.
"I think it's very hard. I think it's a really complicated thing to even think about," he added.
FDA officials have touted the use of the Bayesian approach and adaptive trial designs at previous meetings, including for data extrapolation for pediatric product development (Also see "FDA Encourages Pediatric Master Protocols With Bayesian Approach" - Pink Sheet, 27 Sep, 2016.) In a New England Journal of Medicine article last year, CDER Director Janet Woodcock and LaVange, who was still at FDA at the time, publicized their support for novel clinical trial designs using master protocols. (Also see "Master Protocols Are Both Welcome And Inevitable – US FDA's Woodcock" - Pink Sheet, 6 Jul, 2017.)
But so far, such innovations in clinical trial design have produced more enthusiasm than regulatory results. Woodcock previously admitted that it will likely be a long time before modernization of the clinical trials system can take place. (Also see "Clinical Trial System 'Broken,' But Modernization Long Way Away – Woodcock" - Pink Sheet, 20 Sep, 2017.)
Although the FDA has not formally adopted a Bayesian approach into regulatory decision making, current and former agency officials stressed throughout the meeting that the agency is "open for business" in exploring new avenues for innovative trial designs.
An Obsession With P Values
Several of the experts also agreed that the scientific community has an obsession with the P value as a measure of efficacy, and that it has become deeply ingrained into the mindset of orphan drug development.
"A change has been long needed in how we approach and analyze these things," said Pharmaceutical Research and Manufacturers of America's (PhRMA's) Chief Medical Officer Rich Moscicki in a later panel. "And the necessity with such small sample sizes available to us, the necessity of having to move beyond our obsession with the p value I think is incredibly important to be stated."
Moscicki, who only recently joined PhRMA after serving as deputy director of science operations in CDER, noted that Bayesian methods are centuries old, but only now is the medical community having serious discussions about incorporating them into a regulatory framework. (Also see "Moscicki's Move From FDA To PhRMA About 'Best' Use Of Leadership Skills" - Pink Sheet, 18 Oct, 2017.)
Scott said FDA is a guilty party in facilitating the obsession. "I want to extend an apology on for the collective guilt for my profession that we've traumatized you into thinking not only that p values are the way to measure effectiveness, but even that they may be a way to measure effectiveness," he said.
How To Move Forward
Panelists also offered an array of solutions to help facilitate the shift of the regulatory paradigm.
Moscicki described the "wedding" of Bayesian approaches with registries and natural history studies as "absolutely key to getting as much information as we can when the sample sizes are indeed so small."
Scott spoke of the need for individual case studies, such as "specific individual examples of decisions being made in a principled way, of data being borrowed in a way that makes sense." Individual case studies can then help to provide guidance on best practices, Scott said.
Many speakers throughout the day spoke of the importance of the patient voice, as rare disease patients are in a position to provide valuable data to industry about their experience. For instance, Gianna McMillan, graduate program coordinator at Loyola Marymount University's Bioethics Institute, stressed the importance of leveraging qualitative data from patients to help inform future researchers.
In an interview with the Pink Sheet, Mark McClellan, director of Duke Margolis and a former FDA commissioner, said he feels the paradigm shift is already underway, noting that FDA tries to consider the totality of the evidence in drug review.
"So I think this is a journey that FDA is being already on," McClellan said. "It's being accelerated by these emerging efforts, often patient-sponsored, to bring together lots of natural history data and the capacity for platform studies, and just learning more, and getting more experience about some leading cases for using Bayesian type methods."