Positive Feelings About Negative Controls: US FDA Explores Another Tool For The Real-World Evidence Toolbox
Executive Summary
CDC precedents and PDUFA VII commitments could help advance the robustness of observational studies based on real-world data.
The US FDA is eager to explore negative controls, an emerging statistical approach to addressing unmeasured confounding in observational studies, as part of its wide-ranging efforts to harness the potential of real-world data and real-world evidence.
FDA Center for Drug Evaluation and Research Office of Surveillance and Epidemiology epidemiologist Fang Tian called negative controls a “potential tool to routinely check for evidence of bias with adjustment to ensure high quality of RWE” at a FDA/Duke Margolis Center workshop earlier this month.
Negative controls offer “nice features but some great challenges as well,” Tian cautioned. They are “just one of the approaches that FDA is thinking about in the context of addressing unmeasured confounding in observational studies.”
Nonrandomized, noninterventional study designs reliant on real-world data – observational studies like cohort studies and case-control studies – are vulnerable to biased effect estimates due to unmeasured confounders, or variables that are related to both the exposure and outcome in the study and could account for the observed association.
The problem of unmeasured confounders “is really the most controversial issue in in reporting and interpreting analysis based on observational data,” University of Pennsylvania statistician Eric Tchetgen Tchetgen said.
“We worry about these unmeasured factors, or measure common causes of a treatment outcome, because they can create spurious associations between the treatment and outcome, even if the former does not cause the latter.”
Negative control methods offer “broad utility” in addressing not only unmeasured confounding but other types of bias like information bias and selection bias, Tian said. Negative controls “can be used with various study designs … to detect bias, reduce bias, correct bias, calibrate confidence interval or p-value.”
Negative controls have gained academic traction over the past decade, and CDER has had a “few experiences” with negative controls, she said. At CDER, Tian leads regulatory reviews of RWE/RWD studies, particularly in hematology and oncology.
Those experiences came from “regulatory submissions and internal literature reviews that involved negative controls,” which were used “mainly for detecting unmeasured confounding,” she said.
The negative control analyses seen by CDER were marked by unverified assumptions around topics of non-causality and sharing common confounding structure, and “often lack sufficient interpretation and justification,” she commented.
PDUFA VII Commitment
The workshop fulfilled one element of the FDA’s commitment to explore use of negative controls in RWE under the Prescription Drug User Fee Act VII commitment letter: “By September 30, 2023, FDA will hold a public workshop on use of negative controls for assessing the validity of non-interventional studies of treatment and the proposed Sentinel Initiative projects.”
The PDUFA VII commitment letter foregrounds the potential for negative controls to improve the FDA’s Sentinel safety monitoring initiative, where the agency is working on methodology for Sentinel and CBER’s Biologics Effectiveness and Safety (BEST) Initiative to “improve understanding of robustness evaluations used to address the consistency of RWE with respect to study design, analysis or variable measurement.”
The agency is directed to “develop new methods to support causal inference in Sentinel/BEST that could address product safety questions and advance our understanding of how RWE may be used for studying effectiveness.”
In addition to the public workshop, the FDA committed to initiate two methods development projects by September 30, 2024:
-
“Develop an empirical method to automate the negative control identification process in Sentinel and integrate it into the Sentinel System tools,” and
-
“Develop a method to use a double negative control adjustment to reduce unmeasured confounding in studying effectiveness of vaccines”
A report of the results is due by 30 September 2027.
Assessing The Groundwork
The FDA has already been active in advancing the regulatory science of negative controls, especially through Sentinel. As a demonstration of how negative control analysis can be used, Tian highlighted a study published in American Journal of Epidemiology on 2 February 2022 that used Sentinel data to examine confounding control in estimations of influenza antiviral effectiveness in electronic health plan data, authored by Harvard pharmacoepidemiologist Phyo Htoo and colleagues.
To see if observational studies on oseltamivir use and influenza complications could be influenced by residual confounding, Htoo and fellow researchers examined a health insurance claims database study of oseltamivir and influenza complications (pneumonia, all-cause hospitalization, or antibiotic dispensed).
Using Sentinel, they identified adults diagnosed with influenza who started oseltamivir the same day and those who did not during three influenza seasons. The analysis used a negative control outcome of nonvertebral fractures and a negative control risk period of 61-90 days post-oseltamivir initiation.
Tian is a CDER representative on a collaboration with the University of Maryland on another project to evaluate the utility of negative controls in drug safety and effectiveness studies using real-world data.
UMd School of Pharmacy’s Zafar Zafari presented the first phase of the research, a “scoping review of the state of use, application and utility of negative controls in pharmacoepidemiologic studies” that looked at 184 studies published through September 2020. The earliest study was published in 2006, with a marked increase in publications since 2017.
The most common data source – “by a mile,” Zafari said – was healthcare administrative data, followed by electronic health records. A majority of studies used negative control outcomes, or variables known not to be causally affected by the study treatment, with fewer using negative control exposures, or variables known not to causally affect the outcome of interest.
Only 34 studies employed both types of negative controls.
The “primary use of negative controls has been to detect bias in the literature,” he reported: 149 (81% ) of the studies in the review used negative controls to assess potential bias in causal associations, “testing the negative control association against the null hypothesis.” The presence of bias was detected in 63 (42%).
Bias correction was a less common use of negative controls in the FDA/CERSI review, employed by only 16 studies to obtain bias adjustment estimates (point estimates or confidence intervals). However, Zafari pointed out, “14 of 16 were published between 2017 and 2020.”
CDC Milestone For Negative Controls
FDA Center for Biologics Evaluation and Research Office of Vaccines Research and Review associate director for novel clinical investigations Hector Izurieta described CBER’s experience using negative outcomes to account for the effects of unmeasured variables in vaccine effectiveness studies. The project, which started approximately six years ago, involved researchers from CBER, Acumen LLC and CMS who analyzed studies using negative endpoints that assessed herpes zoster vaccine effectiveness and influenza vaccine comparative effectiveness.
The influenza study resulted in a milestone for negative controls when the Centers for Disease Control and Prevention’s Advisory Committee on Immunization Practices cited it among data sources used to justify a preferential recommendation for the elderly at its 23 June 2022 meeting, Izurieta said. The ACIP recommendation was the first significant regulatory action to explicitly credit a study using negative controls.
The study, which was published in Clinical Infectious Diseases in 2020, compared the effectiveness of all five influenza vaccines used in the elderly using data from more than 12 million vaccinated Medicare beneficiaries.
“We used multiple methods to account for unmeasured confounding associated with what we considered are important sets of potential cofounding factors unmeasured in claims databases,” Izurieta summarized, including physical fitness to care for self and other frailty measures, level of education and health-seeking behaviors.
The project’s earlier claims-based Zostavax analysis piloted the approach of using the nationally representative Medicare Current Beneficiary Survey to evaluate cohort balance in light of unmeasured confounders.
The researchers identified negative outcomes – independent medical encounters that would be expected to show up in claims data for the population but are unlikely to be associated with the intervention of interest – and looked to see how the standardized mean differences (SMDs) aligned between vaccinated and unvaccinated patients.
The influenza vaccine comparative effectiveness study used a set of seven health-seeking behavior indicators: cataracts, eyelid disorders, hemorrhoids, ingrown nail, lipomas, UTI, and wound of hand or finger.
The distribution of the health-seeking behavior indicators across the five vaccine cohorts was very close, and the SMDs for all the associated negative endpoints used to assess of unmeasured confounding “were, reassuringly, very low,” Izurieta said.
The research group next plans to use “double negative controls, including focusing on health seeking behavior bias, in claims studies of vaccines for influenza and other vaccine preventable diseases,” Izurieta said. The seven health-seeking behavior indicators from the prior studies are suggested as negative outcomes, but the double negative control design also requires negative health-seeking behavior exposures. Izurieta suggested breast cancer and prostate cancer screening as negative exposures.
Proposed Projects: DANCE-ing With Double Negatives
FDA CBER Office of Biostatistics and Pharmacovigilance mathematical statistician Yun Lu presented a proposed evaluation of double negative control adjustment for real-world vaccine effectiveness studies. Such real-world studies can have huge sample sizes and can reflect exposures and outcomes during routine clinical practice, but subjects have a wider range of health conditions that in randomized controlled trials.
The FDA has been cautious in its assessment of the comprehensiveness of real-world safety and effectiveness studies of vaccines. Lu pointed to the FDA’s September 2021 draft guidance on electronic health record and claims data, which states that “in general, EHR and medical claims data do not systematically capture the use of nonprescription drugs or drugs that are not reimbursed under health plans, or immunizations offered in the workplace.”
The mass vaccination efforts early in the rollout of COVID-19 vaccines often did not collect insurance information, Lu emphasized.
The proposed vaccine effectiveness study in Lu’s presentation would focus on herpes zoster vaccine exposure. Zostavax and Shingrix are costly enough that out-of-pocket payment is unlikely, and zoster vaccines are not usually offered at mass vaccination or workplace campaigns, so the risk of exposures that would not be captured in medical claims data is low, she explained.
Two previously published studies have measured vaccine effectiveness and duration of effectiveness for Merck’s Zostavax and GSK’s Shingrix.
The study would look at outcomes of different severity that are reliably captured by claims data: herpes zoster infection, hospitalization for herpes zoster, and post-herpetic neuralgia. The unmeasured confounder at issue would be health-seeking behavior. “Vaccinated individuals may tend to seek more health care than unvaccinated officials,” Lu commented.
The study team now needs to identify appropriate negative control exposures and negative control outcomes, Lu said, adding: “we are open to suggestions from the audience!”
“The identification of controls … is obviously the first step in using these tools,” Richard Wyss, Harvard Medical School, said. “Typically, negative control variables must be identified laboriously from background knowledge,” he noted, “and it also has to be assumed that the identified variables were genuine negative controls as no validation test existed.”
While this process has worked well in many studies, it faces limitations, particularly in settings where little background information can be known, such as when a new drug comes on to the market.
To supplement the expert selection of negative controls in such settings, researchers led by University of Michigan biostatistician Xu Shi and University of Minnesota’s Erich Kummerfeld developed a validation test focused on discovery of negative controls “of a special type – disconnected negative controls – that can serve as surrogates of the unmeasured confounder,” Wyss reported.
“While negative controls can be causally related to one of the treatment and outcome,” he explained, a disconnected negative control is “a negative control that is causally related to neither the treatment nor the outcome.”
The researchers developed an automated test to find disconnected negative controls, the “Data-driven Automated Negative Control Estimation (DANCE) algorithm.” The DANCE algorithm has been published, and is now the focus of a study to “extend, test and adapt the DANCE algorithm to large-scale healthcare data reflective of Sentinel data environments,” Wyss said, because “little is known about how DANCE performs in the setting of large healthcare sources where thousands of variables could be considered as candidate negative controls.”
Sentinel Innovation Center data assets, which include linked claims with EHR data structures, “provide an ideal environment to investigate and adapt the DANCE algorithm for practical use,” he stated.
The first phase of the study will use the Mass General Brigham (MGB) Research Patient Data Registry, which includes the electronic health records of all patients aged 65 and older in the MGB registry linked to Medicare claims data. Plasmode simulations will be constructed using two cohort studies generated from the linked data.
“We will design a sequence of plasmode simulation studies based on the empirical example with a range of tightly controlled parameters and known truth,” Wyss said, and “we will simulate true disconnected negative controls to evaluate DANCE’s ability to identify them, and we will control the true value of the causal effect to evaluate DANCE’s ability to estimate it.”
“The simulation studies will allow us to adapt and tailor DANCE to the unique challenges when dealing with complex and large health databases,” he explained.
Phase II of the study will continue with actual Sentinel data linked to commercial EHR-claims from a commercial network as a prototype evaluation. Phase III will develop an ARIA tool and fully checked codes with documentation for data partners in Sentinel.