Developing a common benefit-risk assessment methodology for medicines - a progress report
This article was originally published in SRA
Stuart Walker, Neil McAuslane and Lawrence Liberti report from a high-level workshop on the current status of the various programmes to develop a common framework for evaluating the benefits and risks of medicines.
Worldwide endeavours to develop a common framework for assessing the benefits and risks of medicines are producing an influx of new material on this important topic.
The regulators and companies constructing a formal framework and methodologies that will allow them to make benefit-risk evaluations in a transparent and predictable manner are moving closer to identifying the tools and features necessary for an effective approach.
The stakeholders are increasingly recognising the need to involve patients more extensively and earlier on in the decision-making process. They are finding that benefit-risk assessment will have to be extended as early and as late as possible in the development continuum and that benefits should be examined as closely as harms. They are also homing in on what types of visualisation tools should be used to describe benefits and risks, thereby enabling stakeholders to better understand and articulate the benefit-risk decision.
These and other discoveries on how to standardise benefit-risk methodologies were revealed at a recent high-level workshop in Washington, DC1. The workshop, hosted by the Centre for Innovation in Regulatory Science, heard discussions on the progress made since 20102 by the different groups working towards defining and implementing a benefit-risk framework within their organisations. The groups behind the key initiatives that presented at the workshop included the US Food and Drug Administration, the European Medicines Agency, the Benefit-Risk Action Team (BRAT) of the US pharmaceutical industry association, PhRMA, and a consortium of four agencies – Health Canada, Swissmedic, Australia’s Therapeutic Goods Administration and Singapore’s Health Science Authority – that is being facilitated by CIRS.
Also revealed at the workshop was the establishment by CIRS of a benefit-risk taskforce that includes representatives from seven regulatory agencies and from industry. The CIRS Benefit-Risk Taskforce aims to facilitate knowledge exchange in the area of the benefit-risk assessment of medicines, thereby enabling productivity and avoiding duplicative efforts. Specifically, the taskforce will engage in the exchange of information, reports and published papers to relevant parties to ensure effective knowledge sharing from the various benefit-risk assessment initiatives. Additionally, the group will make recommendations on proposals for workshops, surveys or research that are needed to develop the appropriate toolbox for benefit-risk assessment and to determine how these initiatives might be integrated to ensure their timely development.
FDA status and plans
Regarding efforts by US regulators to standardise benefit-risk assessment, Theresa Mullin, director of the Office of Planning and Informatics at the FDA's Center for Drug Evaluation and Research, told workshop delegates about the benefit-risk assessment template the agency is developing (see Figure 1). A key message from Dr Mullin’s presentation was that the voice of the patient will be increasingly important in the coming years as a component of assessing the benefits and harms associated with a new therapy.
The FDA's benefit-risk framework is being designed to allow the review team to summarise succinctly in a few pages the key points of a much larger documentation of the review of a new medicine, according to Dr Mullin. It is intended to function as a facilitator of internal dialogue among reviewers and to facilitate public communication about FDA decisions.
The framework has been piloted using case studies of previous regulatory decisions and facilitates the evaluation of the benefit and risks of a new medicine using the following five parameters: analysis of the condition; unmet medical need; clinical benefit; risk; and risk management.
Analysis of the condition assesses the potential morbidity and mortality of the untreated disease or condition. Unmet medical need describes the benefits and potential harms of currently available therapies for the disease or condition, including consideration of untreated subpopulations, an area where patient input can be valuable. Clinical benefit evaluates data supporting the efficacy of the drug in terms of decreased morbidity or mortality or the alleviation of symptoms. Risk considers the occurrence of adverse events associated with the drug, including their frequency, severity and reversibility. Finally, risk management documents the sponsor’s plans for post-marketing studies and other measures designed to ensure the minimisation of potential harm to treated populations.
In addition to evaluating available evidence, that is, that which is explicitly known regarding a new medicine, including the submission data from controlled clinical trials and disease state and therapeutic class information from published literature, the framework has evolved for the enhanced consideration of the uncertainties that may surround a drug. These unknown factors include the drug’s potential use and misuse in larger populations as yet unstudied in clinical trials.
Concerning the need to include patients more extensively in the decision-making process, Dr Mullin explained that there is diversity in patient perspective and variation in risk tolerance even among patients with the same disease. She said that the FDA planned to convene public periodic meetings to engage patients, physicians, researchers and industry members with interest in each of approximately 20 specific serious disease areas. It is anticipated that discussions at these meetings will address measuring disease severity, the merits and flaws of current therapies and unmet medical needs, including underserved populations. Industry resources and support are being sought for the development of reliable, validated patient-reported outcome tools that can form part of a comprehensive development strategy, Dr Mullin said.
The FDA also plans to systematically and consistently evaluate the effectiveness of various risk management strategies in use by industry to enhance the information that may be available for a particular patient population for a specific disease at the time of regulatory review.
Also, the agency plans to press industry for electronic submission and for the standardisation of data and terminology for purposes of easier input into modelling tools for analysis and for the quantification of benefits and risks and the characterisation of their associated uncertainties. These efforts will support FDA efforts to optimise the transparent and explicit nature of its decision making, Dr Mullin said.
Progress by the EMA
Progress on the EMA's five-part benefit-risk methodology project, which was initiated in 2009, was provided by Dr Lawrence Phillips of the London School of Economics and Political Science. Work packages 1and 2 have been completed. WP 1 showed that all of the five agencies assessed arrived at benefit-risk decisions “intuitively” through a process of discussion; none of them used a formal model for benefit-risk assessment. WP 2 reviewed the applicability of current tools and methods for benefit-risk evaluation.
WP 3, which involves the adaptation and field testing of recognised tools and processes (from WP 2), is under way. WPs 4 and 5 – the development of a benefit-risk tool/method that can add value in the regulatory process and the development of a training package on the new tool/method for regulatory assessors, respectively – are under development.
EMA versus US frameworks
The EMA and FDA benefit-risk frameworks under construction are conceptually similar in terms of the elements being considered for inclusion, even though they will be subject to different legislation. Dr Mullin noted that, despite the similarities, cultural differences underpin the frameworks (such as variations in the acceptability of risk among European and US populations) as do practical differences in the nature of the data received by each agency upon which to form their decision (ie the availability of patient-level data to the FDA), These differences may, in part, underscore the divergences in approvals that sometimes occur between the two agencies rather than represent a difference in the underlying approach to scientific assessment, Dr Mullin said.
Update on PhRMA BRAT
Workshop delegates received an update on the BRAT framework that has been developed by PhRMA. The framework, which PhRMA started working on in 2006, comprises a set of flexible principle guidelines and tools designed to help a decision-maker select, organise, understand and eventually summarise the key data relevant to a particular benefit-risk decision and promote the transparency of the decision making process, said Diana Hughes, vice president, worldwide safety strategy, primary care business unit lead, Pfizer, US. Although the framework focuses on qualitative elements, it has the ability to incorporate any available quantitative methodologies necessary to introduce weighting based on conjoint or other analyses, Dr Hughes added.
The BRAT framework is being tested in a pilot study called the Soft Pilot Programme (see Figure 2). The pilot, which started in February 2011, had enrolled ten companies by mid-2011 and is in the implementation phase. In November and December 2011, post-pilot assessment interviews will be conducted to determine individual contexts in which the framework was applied. With the ultimate goal of framework refinement based on input from participants, a second, online post-pilot assessment is also planned to gather additional information regarding the effectiveness and use of the framework, Dr Hughes said.
The four-agency consortium
To facilitate the opportunity for shared or joint review involving Canada, Switzerland, Australia and Singapore, a consortium of medicines regulators from these countries has been working on a qualitative benefit-risk framework since 2009.
The consortium, which is working with CIRS, has developed a draft framework or “proforma” for documenting the key information for benefit-risk assessment. The consortium expects to accrue specific benefits through the use of a benefit-risk template, Supriya Sharma, (at the time, director general, Therapeutic Products Directorate, Health Canada), reported at the workshop. Although certain topics such as the weighting of risks and benefits and visualisation methods are still under discussion, a retrospective pilot study of the framework is planned to be completed by the end of 2011.
With regard to the issues that are still to be determined, Dr Sharma said that the consortium members would have to decide whether the finalised template would replace existing documents at their agencies or function as an alternative or additional document for purposes of work sharing among the group. She also noted that, when complete, the template is expected to be large and comprehensive, and the amount of time that may be needed to complete it could present an obstacle to reviewers with timelines that are already compressed.
The need for continuous monitoring
The need to extend benefit-risk assessment from early on in the product lifecycle to beyond the post-marketing stage was discussed in three presentations at the workshop.
Ellen Strahlman, chief medical officer of GlaxoSmithKline, US, noted that several recent studies have found an association between the GSK anti-epilepsy drug lamotrigine and sudden unexplained death from epilepsy. While the results of these studies are not conclusive, the emergence of these data almost 20 years after the medicine’s approval points to the fact that benefit-risk evaluation should extend far beyond obligatory post-marketing milestones, Dr Strahlman said.
Alain Micaleff, senior medical safety advisor of Merck Serono SA, Switzerland, reported on a European project that is developing methods for continuous benefit-risk monitoring of medicines. PROTECT (Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium), a five-year project that began in 2009, is being run by the European public-private partnership, the Innovative Medicines Initiative.
Work package 5 of the seven-package project aims to integrate data on benefits and risks from clinical trials, observational studies and spontaneous post-approval reports, including the underpinning, modelling and the presentation of the results, with a particular emphasis on graphical methods, explained Dr Micaleff. It covers only the benefit-risk assessment during the submission and post-approval periods. The project addresses individual and population-based decision-making and considers the perspectives of multiple stakeholders including patients, physicians, regulators, industry and the societal views necessary for health technology assessment.
Dr Micaleff said that WP 5 would incorporate the review and selection of methodologies and visualisation methods, the choice and implementation of case studies and the visualisation and communication of benefit-risk.
Mark Walderhaug, associate office director for risk assessment at the FDA's Center for Biologics Evaluation and Research, noted that the understanding of risks and benefits of a new medicine evolves over time. Together with his colleagues, Dr Walderhaug has used an integrated computing system, called Mathematica, to develop a theoretical model that represents the benefit-risk-uncertainty profile of a medical product, integrating and unifying data on multiple properties and allowing the visual comparison of the benefits and risks of two products.
Dr Walderhaug provided the workshop delegates with a realistic but fictionalised example of the ability of the model to map the benefit-risk evolution with the case of a hypothetical vaccine to protect against a common virus that causes severe diarrhoea in children. In this hypothetical case, the benefits of the vaccine’s use in developed countries included reduced disease burden and hospitalisation for children and reduced burdens on caregivers; the primary risk was intussusceptions, a serious and potentially life-threatening condition in which the intestine becomes blocked or twisted. Because the occurrence of this adverse event is very rare, it was not discovered during clinical trials. Using binomial distribution to simulate the occurrence of adverse events, the model was able to show no increased risk for the use of a second vaccine dose using the assumptions that vaccines prevented an average of 91.5% (85%-98%) of hospitalisations and 1 in 15,000 children receiving the first vaccine had a severe adverse event caused by the vaccine.
Visualisation tools for communicating benefit-risk decisions to consumers and other healthcare stakeholders were described in several workshop presentations.
James Felli, a research fellow at Eli Lilly in the US, talked about the power of visualisation tools for integrating benefit and risk data and communicating the implications of these trade-offs. Representing the unfavourable and favourable effects of a drug as hues of colour is one example of a visual method for allowing a stakeholder to assess the degree of benefit and harm and, therefore, be more informed about selecting the trade off between risk and benefit with which they are most comfortable, Dr Felli said.
Sweden's Medical Products Agency produces national product monographs containing unbiased information regarding new medicines that are used by health technology assessors and the health authorities to inform therapeutic and reimbursement decisions. Jane Ahlqvist-Rastad, senior expert at the MPA, presented case studies of two MPA product monographs that demonstrated negative benefit-risk balances and the consequences of those evaluations. Dr Ahlqvist-Rastad said that although healthcare professionals in Sweden are now accustomed to receiving this information in the form of a monograph, the monographs could be improved by highlighting uncertainties and using methods for visualisation such as benefit-risk value trees.
To learn about the ways in which benefit-risk is communicated to patients and healthcare professionals, Sam Salek, professor of pharmacoepidemiology, Cardiff University, posted a Benefit-Risk Communication Index developed by his team to a random sample of physicians, nurses and pharmacists. More than half of the 1,167 responding healthcare professionals felt that a more systematic approach to risk communication by the government and companies was needed (see Figure 3).
Professor Salek called for additional research and training in the communication of benefit-risk to healthcare professionals and by healthcare professionals to their patients. He urged the government and industry to adopt a systematic approach to risk communication. The provision of benefit-risk information in an understandable manner as part of patient information leaflets would be highly beneficial, as would the involvement of patients and healthcare professionals in the development of relevant information, Professor Salek concluded.
Including patients in decisions
The importance of including patients’ perspectives in the decision making process was also highlighted in a session at the workshop in which participants were invited to test benefit-risk assessment methodologies using a hypothetical drug. The delegates found that using either qualitative or quantitative methodologies, a decision process that was informed by the patient’s perspective could reach a consistent conclusion whose rationale could be communicated clearly and transparently to other stakeholders.
Overall conclusions from the session were as follows:
assessment tools, be they more qualitative or quantitative in their approach, help provide a transparent structure for the discussion of multi-factorial elements of benefit-risk assessment of a medicine;
visualisation tools help to focus benefit-risk discussions on critical issues, identifying gaps and exposing overlapping benefits and harms and providing a succinct summary of the information needed to make benefit-risk decisions;
in addition to transparency, consistency and communication, articulating and reaching consensus on research questions and meticulously delineating target populations are essential to benefit-risk evaluation;
for conditions involving subjective benefits and harms, patient input is invaluable in informing the thinking of decision makers such as regulators and researchers; and
communicating decision rationales reached through multi-criteria decision analysis to prescribing physicians may require some education on methodology. Results could be simplified for patients by providing a graphically displayed quantification of trade-offs.
While workshop delegates agreed that a standardised framework and related methodologies for benefit-risk assessment were necessary, there are challenges to their development, including international and regional differences in available tools, the rigour with which these tools are used, the limitations imposed by jurisdictional regulatory processes and the complexity of addressing diverse clinical guidelines and labelling requirements.
Results from a CIRS survey of 20 companies and 11 agencies conducted to gauge the status of the development of benefit-risk systems, criteria for their use and the advantages and disadvantages of the various models were presented at the workshop. The survey, described by CIRS director Neil McAuslane, also aimed to identify internal and external barriers and possible solutions to the incorporation of a benefit-risk framework and methodologies into the development, regulatory review and ongoing assessment of medicines.
Although none of the companies or agencies that responded to the survey is currently using a fully quantitative model for benefit-risk assessment, agencies and companies are both looking to improve on their existing methodologies or systems of assessment and some of those using qualitative frameworks are looking to incorporate elements of a semi-quantitative or quantitative model, Dr McAuslane reported. In all likelihood, he said, the methodology(ies) that an organisation will settle on will incorporate elements of both qualitative and quantitative decision making.
The survey showed that the key perceived advantages for implementing a benefit-risk framework, particularly for companies, were as a tool for communication, structured discussion and enhanced transparency and accountability. On the other hand, the biggest barrier named was a lack of a scientifically accepted or recognised framework or methodologies. There is generally good agreement, however, between agencies and companies on the need and function of an appropriate benefit-risk framework. Also, the key hurdles to its implementation mainly relate to the divergent stakeholder perspectives and lack of accepted methodologies and solutions. Finally, development of visualisation tools for communicating benefit-risk balance within companies and agencies seems to be in its earliest stages.
Another ongoing issue among benefit-risk stakeholders highlighted at the workshop was the unequal concentration that is placed on monitoring potential risks associated with a medicine at the expense of considering its benefits, especially in the post-approval period. Dr Strahlman said it would be advantageous for stakeholders to improve their "willingness and ability" to track the impact of a medicine’s benefits over time and evaluate long-term changes in the benefit picture. Over the past 20 years, treatments for cardiovascular and HIV disease, cancer, rheumatoid arthritis and diabetes have had a tremendous positive impact, turning formerly fatal conditions into chronic but treatable illnesses, she pointed out. By applying a benefit-risk assessment to an evaluation of these therapies, it may be possible to learn from past examples of disease area benefits.
In conclusion, Hans-Georg Eichler, senior medical officer at the EMA, said that benefit-risk methodology and presentation is currently evolving to that CIRS-stated goal of “an internationally accepted, systematic, routine and standardised documentation of [benefit-risk] decisions". It is likely that this continued evolution will incorporate both qualitative and quantitative assessments moving from implicit to explicit value judgements, adding to regulator’s values those of the patients, Professor Eichler predicted.
In the future, benefit-risk decisions will be graphically demonstrable in a patient-understandable way. A toolkit for benefit-risk assessment must be developed that will make regulatory decisions predictable and auditable, with differing models for individual situations. Using the same detailed and systematic tools should permit the correlation of regulatory opinion regionally, nationally and internationally, Professor Eichler said.
CIRS is planning to hold a follow-up workshop in June 2012 in Washington, DC, entitled: Building the Benefit-Risk Toolbox: Are there enough common elements across the different methodologies to enable a consensus on a scientifically acceptable framework for making benefit-risk decisions?
1. The Centre for Innovation in Regulatory Science workshop, Visualising benefit-risk: The key to developing a framework that informs stakeholder perspective and clarity of decision making, Washington DC, 16-17 June 2011
2. Walker S, McAuslane JN, Liberti L, Refining the benefit-risk framework for assessing medicines: valuing and weighting the parameters, Scrip Regulatory Affairs, 23 February 2011
Stuart Walker is the founder of the Centre for Innovation in Regulatory Science in London, UK. Neil McAuslane is the scientific director of CIRS and Lawrence Liberti is the centre's executive director. Email: firstname.lastname@example.org.