Tocilizumab COVID-19 Study Shows Bias Against Observational Studies, Harvard Prof Says
Executive Summary
Study showing Roche’s Actemra decreased mortality in COVID-19 patients in ICUs by almost 10% was initially rejected by journal editors and then ignored by guideline writers and regulators. FDA officials and industry reps discuss what needs to be done to advance use of real-world evidence from observational studies.
The US Food and Drug Administration is evaluating the potential role of observational studies in demonstrating drug product effectiveness. However, the response to an observational study of Roche Holding AG’s Actemra (tocilizumab) for COVID-19 patients shows the hurdles such studies face in gaining acceptance.
Miguel Hernán, professor of biostatistics and epidemiology at Harvard T.H. Chan School of Public Health, described the study he and his colleagues conducted in 3,942 individuals with COVID-19 in intensive care units in 68 US hospitals. The study found 30-day mortality was 27.5% in those receiving tocilizumab and 37.1% in the group that did not receive tocilizumab.
Speaking at a 16-17 February Duke-Margolis Center for Health Policy meeting on evaluating real-world evidence from observational studies, Hernán reported that the study results were largely ignored by the scientific community.
One journal editor said the publication wanted to wait for actual trials of the drug, “essentially saying we don’t trust observational data,” Hernán said. Another editor expressed concern there was a high risk of placebo confounding, without explaining why there could be such a risk. The study was published in the Journal of Internal Medicine in May 2020, but Hernán said journal editors, treatment guideline writers and regulators ignored the observational analysis.
In fact, he said the use of tocilizumab has not been recommended during much of the pandemic, except maybe in the UK, and the US National Institutes of Health did not mention the observational study in its treatment guideline.
“This was the perfect setting for observational studies that need to be taken seriously. And that’s how biased we are against observational studies. Not even in this situation do we use them,” he stated.
Hernán noted that the results of a large trial were recently published that found the same results of the observational study. That study, the UK-led RECOVERY trial, found that patients given tocilizumab are more likely to be discharged from hospital alive within 28 days compared with those on standard of care (54% vs. 47%). (Also see "More Good News On Actemra In Severe COVID-19 – But Will Roche File For Approval?" - Scrip, 15 Feb, 2021.)
FDA's Duplicate Project
The Duke-Margolis conference provided an overview of projects being conducted to replicate the results of randomized clinical trials (RCTs). Mark McClellan, director of the Margolis Center for Health Policy at Duke University, explained that the goal of these projects is to use observational studies with rigorous design to understand when evidence from these studies may be credible.
The trial replication projects include the FDA-funded RCT DUPLICATE project, in which researchers from Brigham and Women’s Hospital, Harvard Medical School, and the health care data company Aetion, Inc. are using claims databases to try to replicate the results of 30 randomized controlled trials. (Also see "Real-World Data Could Get Boost From Trial Replication Project" - Pink Sheet, 26 Apr, 2018.)
FDA’s Jacqueline Corrigan-Curay, director of the Office of Medical Policy in the Center for Drug Evaluation and Research, said the goal of the project is to better understand whether there are "study designs that could reach the same regulatory conclusions about these products,” Corrigan-Curay said.
One of the study investigators, Sebastian Schneeweiss, Harvard Medical School, presented a summary of the results of the first 10 replication studies, which were published in December in the American Heart Association’s journal Circulation. He noted that the Duplicte project is seeking to replicate 30 RCTs and predict seven ongoing Phase IV RCTs. He said he wanted to learn whether FDA would have come to the same regulatory decisions if an RCT that was submitted, or could have been submitted to the agency, was replaced with a single real-world evidence study.
Eight of the first 10 randomized trials involve antidiabetic medications. Three of the ten RCTs were active-controlled and seven were placebo-controlled. The investigators found that despite attempts to emulate RCT design as closely as possible, differences remained between the RCT and corresponding RWE study populations. The journal publication notes that the regulatory conclusions were equivalent in six of ten studies. The largest differences in effect estimates were found for RCTs where second-generation sulfonylureas were used as a proxy for placebo regarding cardiovascular events.
"The most important outcome may very well be that you have a repository of valid documented studies that illustrate the agreement between RCTs and real-world evidence, and then in specific situations where the real-world evidence study is explicitly designed to answer the same question as the RCT," Schneeweiss said.
Sample Characteristics May Differ
William Crown, a research scientist at Brandeis University, reported the results of the Operand trial emulation project, which replicated two trials, the ROCKET Atrial Fibrillation Trial comparing rivaroxaban once daily with vitamin K for prevention of stroke and embolism, and the LEAD-2 trial comparing liraglutide/metformin versus metformin monotherapy versus metformin/glimepridine in treatment of type 2 diabetes. He said researchers will soon submit the findings for peer review.
The study sought to determine whether observational studies using real-world evidence replicate RCTs submitted for regulatory decision-making, to develop empirical data to understand data quality and limitations of real-world data from various data sources, including claims data and electronic medical records, and to determine how RWE informs understanding of drug effectiveness in approved populations.
Crown, who co-led the project, said the sample sizes and characteristics were very similar in the ROCKET trial and observational study but varied substantially between the LEAD-2 trial and observational study. He said for both trials, there was very little variation in the treatment effect estimates when the inclusion/exclusion criteria were loosened so the results are not generalizable to all other disease states or different clinical interventions.
"What have we learned from these emulation efforts?" he asked. "I think when we're doing the design of the target trial for when we have an actual trial to attempt to emulate, we can still have results that may dffier in important ways in design of that target trial from the actual trial."
However, he noted that the growing literature on emulation efforts shows it is often possible to arrive at very similar effect estimates with observational data. He said that in these emulations, sample characteristics of observational data sometimes look quite differnt from those of the randomized process. "That's worrisome, but not necessarily fatal," he said.
When Clinical Trials Do Not Align Real World Practice
Nilay Shah, chair of the division of health care delivery research at the Mayo Clinic, discussed three emulation studies being done by the Yale University-Mayo Clinic Center for Excellence in Regulatory Science and Innovation (CERSI), which is funded by the FDA. These include emulations of the PRONOUNCE trial and the GRADE trial, both conducted with FDA, and the CABANA trial, which was emulated as part of an award from the National Heart, Lung, and Blood Institute. Shah said the trials were emulated to predict both the trial population and results of ongoing clinical trials. All three use OptumLabs claims data.
The PRONOUNCE trial is a safety trial that randomaly allocated 900 participants with advanced age, prostate cancer and cardiovascular disease to either degarelix or leuprolide, with the main goal of looking at adverse cardiovascular events. The results of the study are not yet available. Shah said when he and his colleagues used real-world data they were able to identify about 8,000 patients who would be eligible for the PRONOUNCE trial after applying both inclusion and exclusion criteria.
The GRADE trial is a comparative effectiveness trial looking at glycemic reduction approaches in diabetes. The study randomly allocated 5,000 adults with type 2 diabetes to one of four drugs (glimepride, sitagliptin, liraglutide and basal insulin glargine). Looking at the sample size between the four groups, Shah said there was a difference between what the real-world practice does versus what the clinical trial specified. The emulation study dropped the glargine arm due to the small sample size. Shah said the results of the emulation study will be available later this year.
The CABANA trial compared ablation to medication therapy for atrial afibrillation in 2,204 patients. One question was whether trial participants represent patients in every day practice. Shah said when they looked at the eligible population compared to the trial population, the eligible population from the real-world data tended to be older, more likely to be female, and to include a greater number of minority patients.
Shah noted that there is a fair amount of difference in how clinical trials are designed relative to real world practice, especially when there are active comparator arms. In addition, he said real world results and clinical trials will not always align.
"I think we'll have to think about on a case-by-case basis, what does that mean? And how do we better understand that and learn from that when those results don't align?" he said. "In some cases that may be okay. In other cases it may be a fundamental issue in design of the study."
Can There Be Substitute For A Randomized Trial?
Robert Temple, CDER’s deputy center director for clinical science and senior adviser in the immediate office of the Office of New Drugs, was enthusiastic about the replication studies.
“I’m very excited about the prospect of what is going on. I think it’s very important to try to duplicate each one of these controlled trials more than once and to look within the various databases that we have,” he said in a panel discussion on the replication trial results. “But what we really need to know in the long run is where these things actually conceivably could be a substitute for a randomized trial for one reason or another, and where they can’t.”
Temple noted that a lot of the studies looked at cardiac events and coronary artery disease events, which he said are sometimes easy ones to assess since the risks and endpoints are well established. He said it will be interesting to see the results of studies looking at more subjective factors, such as heart failure.
Temple also noted that the agency is working hard to decrease “stupid exclusions from controlled trials,” such as exclusions by age or other diseases.
He noted that for the most part, historical controls are credible when the effect sizes are large. “We already use so-called historical external controls to see if the cancer shrinks, that’s a no brainer. But have you reduced the chance of dying from a heart attack?” he said. “We have some way to go, but it’s an area of interest.”
Summing up the panel discussion, Temple said “whether and when observational data alone, not randomized, can become credible enough to be used is what in part we’ve been talking about."