Refining the benefit-risk framework for assessing medicines: valuing and weighting the parameters
This article was originally published in SRA
Stuart Walker, James Neil McAuslane and Lawrence Liberti report on progress by regulators and industry on the development of a common benefit-risk assessment framework.
A recent survey of drug companies on nearly 40 scenarios that may influence aspects of the regulatory environment by 2020 has indicated that the development of a common framework for the benefit-risk assessment of medicines for use in dossier reviews and in communicating the results of those reviews to stakeholders is a top priority1.
The Project 2020 survey was conducted by the Centre for Innovation in Regulatory Science (CIRS), formerly the CMR International Institute for Regulatory Science. It underscores how there is increasing recognition of the need for a standardised method of assessing benefits and risks.
Determining the benefit-risk balance of a medicine is one of the most important steps in a drug's development, review and post-approval reassessment. A common framework for making such an assessment would be critical from both a regulatory agency and pharmaceutical company perspective in order that there be a transparent articulation of the benefits and risks considered in making the final regulatory decision. It would allow all stakeholders, regardless of their backgrounds or responsibilities, to verbalise their understanding of the value of a particular parameter, thereby providing the context for a structured discussion about each one's decision-making process.
A number of groups, including the CIRS, regulatory agencies and industry are involved in efforts to create benefit-risk frameworks2. Consensus has been achieved on the essential elements that should comprise any framework and on the seven key steps that take data through a systematic process prior to making the final decision. The CIRS says that these steps are: establishing the decision context; developing a value tree for all benefits and risks identified in the submitted dossier; tabulating and providing the rationale for inclusion of benefit and risk parameters by regulator; providing qualitative values for options (investigator drug, comparator, placebo); weighting of benefit and risk parameters; producing a qualitative/quantitative visualisation of benefits and risks; and expert judgment of final benefit-risk decision.
However, while there is agreement concerning the steps that are required for the framework, the assignment of weighting and values to each component parameter is more difficult, owing to different views, methods and stakeholder perspectives, not just on what should be used in weighting, but how it should be used.
Seeking consensus at a Washington workshop
In 2010, the CIRS, a UK-based, not-for-profit organisation that works in the regulatory and policy arena, hosted a workshop in Washington, DC. The aim of the workshop was to develop an agreement on the principles that should be used in the weighting and values step of the framework and to discuss how it could make its own framework more robust, understandable and practical. The forum was attended by experts from regulatory agencies, academia and the pharmaceutical industry. The various approaches to standardising benefit-risk assessment such as that of the CIRS, the European Medicines Agency and the US pharma industry association, PhRMA, have a common genesis in decision analysis. Today they are converging to draw on each one's strengths as efforts build towards the common goal of developing a standardised tool for the transparent assessment and discussion of a medicine's benefits and risks.
However, while a number of agencies are working collaboratively at present, there are challenges to the agreement of a common approach. In Europe, for example, there are 27 member states and each of those may have a different perspective, a different approach and a different way of making benefit-risk decisions. Indeed, they may even have different definitions of what they understand by benefits and harms. Both the CIRS and the EMA are addressing the publication of worked examples of different methodologies of benefit-risk assessment. In addition, the EMA is looking across different methodologies and approaches and systems in order to make a side-by-side comparison.
The workshop kicked off with a presentation by Stuart Walker, founder of the CIRS. Professor Walker described the evolution of the CIRS's benefit-risk framework over the past ten years. He explained that this framework – which has been endorsed by international regulatory agencies and has served as the conceptual underpinning for other initiatives in this arena – is well designed to permit the integration of a weighting/valuing system as a critical step to support final expert judgment and decision-making.
The CIRS's framework uses a qualitative approach, which it says is the most appropriate way forward at this stage, although it hopes to develop it into quantitative methodologies in the future. The framework is based on a seven-step decision-making process outlined in the book "Benefit-Risk Appraisal of Medicines: A systematic approach to decision-making", by Mussen, Salek & Walker, which is as follows3:
look at the context of the decision;
identify the options, which in the case of a new medicine is the new medicine, the comparator, and the placebo;
identify the benefit and risk criteria;
organise those into a value tree;
assess the performance of each of these options against the criteria and calculate the uncertainty;
assign a weight to each criteria; and
produce a weighted score, and conduct a sensitivity analysis.
In the most critical step, a valuation is made of the product versus an active comparator or placebo, in some subjective, qualitative, or in some cases, quantitative way, resulting in the expert judgment of benefit-risk assessment (see Figure 1). This judgment relies on wisdom and experience as well as the data that have been provided in the submission. Professor Walker pointed out that this is a somewhat challenging methodology, which may not be feasible for use by regulatory authorities at this stage of development.
Professor Walker also discussed the CIRS's recent survey of benefit and risk systems. The centre's member companies had been questioned regarding their use of a formal systematic structured benefit-risk framework, the hurdles to establishing the framework and solutions to overcome those hurdles. The following six basic hurdles have emerged:
the lack of an accepted, validated, international model that the US Food and Drug Administration, the EMA and the Japanese Pharmaceuticals and Medical Devices Agency would recognise and use in their assessment of a new drug application;
the absence of an academic, industry or government agreement on methods for the quantitative valuation of benefits and risks;
perceived difficulties in making and applying value judgments;
a reluctance to change from existing assessment methods, tempered by cultural issues;
an insufficient internal belief at the level of senior R&D staff of the added value of a benefit-risk framework; and
the need to use a thoughtful, cautious approach towards moving to complete decision-making transparency .
Solutions to these perceived hurdles were also suggested by the respondents. The respondents called for:
the establishment of a collaborative working group of representatives from industry, regulators, patient groups and healthcare professionals;
an agreement between industry and regulatory agencies on a common approach, or at least on a common set of principles for a benefit-risk framework;
internal education on the value and approach to the quantification of benefits and risks;
the identification of the characteristics of a benefit-risk model recognised by the FDA and other health agencies;
benefit-risk methodology to be taught in medical and pharmacy schools to prepare the next generation of medical reviewers; and
the availability and publication of worked examples using different methodologies of benefit-risk assessment, as the basis for guidance.
The CIRS has now developed a proforma with an electronic version planned for the documentation of the benefit-risk decision-making process based on the seven steps outlined above. It is hoped that a pilot study will be carried out on products being evaluated by the four agencies that have formed a consortium, namely the Therapeutic Goods Administration in Australia, Health Canada, the Health Sciences Authority in Singapore and Swissmedic, during 2011.
Several industry executives gave presentations on their companies' efforts to improve benefit-risk analysis.
David Jefferys, of Eisai Europe, said that his firm had introduced Global Regulatory Analysis Strategic Plans (GRASP), electronic records of each new medicine's complete global strategy. Available company wide, Dr Jefferys said that GRASP has proved a valuable source of cross fertilisation. Eisai has found that benefit-risk assessment also improves internal communications, helping groups to focus on key product attributes.
Dr Jefferys noted that although it is widely recognised that benefit-risk evaluation is a work in progress, one of the issues that is emerging is how the evaluation can be incorporated into a dossier, especially into a global regulatory dossier. Related questions include: where the assessment should be placed in the dossier; how the balance should be presented between benefits and risks in the assessment; and ways to address the different risk management plans. In today's international environment, there is a need for a comprehensive, global approach, but there are different requirements emerging between approaches taken by the EMA and the FDA, and the PMDA is now entering into the discussion. These disparate regional perceptions regarding both risk and benefit need to be addressed when making simultaneous global filings.
From Novo Nordisk A/S, Drs Sinan Bardakcki Sarac and Christine Hallgreen described a simple and transparent method their company used to assess and analyse the treatment-related effects of drugs in clinical trials while preserving as much objectivity as possible for decision makers. The Benefit Risk Assessment in New and old drugs (BRAIN) method can extract information from clinical trials, which are otherwise not captured by statistics. It consists of the following eight steps:
Step 1: Decision context – defines the aims and goals of the assessment.
Step 2: Disease profile – includes the identification of benefit and risk criteria that characterise the disease. These criteria are either measured objectively, eg blood pressure, or subjectively, eg quality of life. Once the most important criteria are selected within the given decision context, one has to justify the choice of these criteria so decisions can be tracked.
Step 3: Weighting – to compare benefits and risks on the same scale, they are weighted on the same value scale. The weights are based on the relative importance of a criterion in the given context and all weights are justified. Weighting is not based on the actual data sets. Each benefit and risk criterion is given a weight of 1 (low), 2 (medium) or 3 (high) importance. The weights are common for all drugs in the given assessment.
Step 4: Scoring – an assessment is carried out of the performance of a drug relative to a comparator by assigning a numerical value for each criterion. Scoring is based on available data sets from clinical trials or other information, eg preclinical findings. For each criterion, the drug is scored relative to the comparator on a simple scale; -1 (inferior), 0 (non-inferior) and +1 (superior).
Step 5: Evidence evaluation – if the evidence is weak an objective score can be changed to an interval (-1 to 0, 0 to +1 or -1 to +1). A list of elements of evidence evaluation have been recommended by the EMA's scientific committee, the CHMP.
Step 6: Weighted scores – weights and scores are multiplied.
Step 7: Presentation of the results – the weighted scores are visualised through a Tornado-like diagram. Results from multiple trials can be combined to give an overall benefit-risk assessment as seen in Figure 2. Each trial is assigned an impact factor based on its importance.
Step 8: Overall conclusion – the hypothesis formulated in the decision context is either accepted or rejected. Any uncertainty is described, including how that uncertainty impacts the results. Unexpected issues are described, and strategies for further studies are presented. Finally, a recommendation and conclusion is given.
In a presentation on the visualisation of benefit-risk, Douglas Manion of Bristol-Myers Squibb discussed the March 2010 BMS presentation of belatacept data to the US FDA advisory committee. Dr Manion described how each element of the benefit and risk value tree for the drug was transparently quantified using simple graphs that included relative point estimates and confidence intervals. This quantitative graphical approach, he explained, allowed the committee to readily understand belatacept's potentially complicated benefit-risk story.
Dr Rebecca Noel of Eli Lilly and Company provided details of the Lilly Benefit-Risk Assessment Model (BRAM), designed to help focus discussion on the primary elements of risks and benefits for a specific treatment and its alternatives. The model uses multiple attributes because it is not only necessary to discuss the relative importance of benefits and risks when assessing the overall utility of a treatment, but also the trade-offs a decision-maker is willing to make between the various aspects of benefit and risk. Finally, Dr Marilyn Metcalf of GlaxoSmithKline explained that this firm's benefit-risk evaluation team is building a framework for the evaluation of benefit-risk that will answer the needs of multiple stakeholders at different levels, including colleagues, regulators, payers, healthcare providers and patients.
Regulatory agency perspectives
On the regulatory front, Dr Xavier Luria of the EMA discussed the benefit-risk methodology project the agency initiated in 2009.
The EMA initiative, in which all 27 EU countries are participating, is expected to improve the quality, transparency and consistency of benefit-risk assessments, resulting in more auditable and robust evaluations. It is envisioned that benefit-risk assessment could be harmonised across the European network and that the time to approval of a market authorisation application could theoretically be reduced.
The project comprises five work packages: firstly, a description of the current practices; secondly, applicability of tools and methods; thirdly, a field test of some of the methods; fourthly, the development of tools and methods for benefit-risk assessment; and finally, a training module for benefit-risk assessors.
The first two work packages have been concluded and published. Work package 1 resulted in the definition of five criteria for assessing the acceptability of benefit-risk methodology and tools: logical soundness, comprehensiveness, acceptability of results, practicality and "generativeness". The results from work package 2 included the identification of the tools and methods to be tested in work package 3. Work package 3 is expected to be completed in the first quarter of 2011. The aim of this work package is to apply live benefit-risk modelling to ongoing procedures. For each participating national agency the assessors' team is engaged in a one-day modelling session. Feedback on the usefulness and applicability of the tested methodology is received by distributing evaluation questionnaires. Four sessions have been completed so far. A report on the work package is expected to be finalised in the second quarter of 2011 and will be published on the agency's website.
Providing the US regulatory perspective, Theresa Mullin of the FDA's Center for Drug Evaluation and Research indicated that the agency was at an early stage of evaluating a potential benefit-risk framework (see Table 1). Dr Mullin noted that the US statutory standard does not require that a new therapeutic product be superior to available choices, only that it be safe and effective for the intended use. This standard implicitly values choices and frames regulatory decision making. Dr Mullin stressed that using a process that clearly outlines the available data and documents how judgments were made can improve transparency of the decision-making process.
The framework, which has been tested in a pilot study, encourages regulators to examine the benefit-risk considerations in the following five dimensions:
the severity of the condition assesses whether the condition will be life-threatening, rapidly fatal, serious, or non-serious if left untreated;
unmet medical need establishes the current state of the armamentarium for a given condition. This includes an assessment of the benefits provided by existing treatments and how well they are tolerated by the patient population. Consideration is also given to underserved patient subpopulations;
clinical benefit addresses efficacy outcome measures, including decreased mortality and whether the product cures the condition or alleviates symptoms;
consideration of risk requires a determination of the frequency, severity, time of onset and reversibility of adverse events. In addition, risk assessment involves considering whether certain subpopulations may be at increased risk for particular adverse events; and
consideration for proposed risk management plans, post-marketing studies, or labelling to mitigate adverse events of concern.
Table 1. Potential qualitative framework from the US FDA
Severity of condition
Unmet medical need
During the pilot phase, several attributes have emerged as desirable qualities in future revisions of the benefit-risk grid. The framework must be "simple" to understand, but avoid "simplistic" judgments. It should support sound expert judgment, rather than acting as a replacement for it. It should identify and respect areas of expert disagreement. In addition, it must support explicit presentation and discussion of the scientific evidence, any uncertainties regarding interpretation of the evidence, implications stemming from analysis of the evidence, and assumptions that are used to aid the process of scientific interpretation or analysis.
Dr Mullin said that it was not clear whether the FDA's benefit-risk assessment would be completely different from what is happening in Europe. There may be different approaches to treatment and different factors in health delivery systems that affect some of those details, she said, noting that differences in the statutory parameters and social expectations would have to be considered.
The FDA official indicated that the agency was being cautious not to move to a more restrictive framework because there is a lot of variability in what may be found in the course of trials. The agency would not, she said, want to make it difficult to include that information.
Another regulatory perspective was presented at the CIRS's Washington, DC, workshop by Petra Dörr, of Swissmedic, and Jason Ferla, of Australia's TGA. Drs Dörr and Ferla detailed progress on the benefit-risk framework by the Consortium, a group formed in 2006 that comprises the Swiss and Australian agencies and also those of Canada and Singapore.
The Consortium together with the CIRS has embarked on a project to develop a qualitative framework for the benefit-risk assessment of medicines that would allow a systematic standardised approach to the appraisal of medicines during the regulatory review and post-marketing by the four agencies. The project was begun in April 2009 with a feasibility study in which a draft benefit-risk template was developed using the value tree developed as detailed in Professor Walker's presentation.
The group has developed a harmonised benefit-risk assessment template. Benefits of using the template have already emerged in a feasibility study. Also discussed at the CIRS workshop were: the role of the template from a regulatory agency perspective; whether the Consortium needs to consider adding graphic representation such as Forest plots to the template; whether a common set of terminology and definitions is required; and how many secondary endpoints should be considered.
The Consortium has developed a proforma for documenting the key information for benefit-risk assessment based on the framework developed by the CIRS. The Consortium is in the final stages of turning the proforma into an electronic document. It plans to pilot test the proforma during 2011 and begin a major prospective study in 2012.
Lessons learnt from the CIRS workshop on refining the benefit-risk framework for assessing medicines were that:
benefit-risk assessment exercises involving all stakeholders should be conducted throughout a product life-cycle with benefits and risks added and subtracted as data become available;
although there should be some overlap in stakeholder opinions regarding the importance or inclusion of specific benefits and risks in the assessment, significant differences should also be expected based on the difference in stakeholder perspectives and should form the basis of stakeholder discussions;
tools such as a value tree and supportive data tables are necessary for a structured benefit-risk debate;
the introduction of novel visualisation tools to diverse stakeholders (ie regulators, clinicians and patients) should proceed in a methodical, educational manner to allow each group to familiarise itself with the strengths and weaknesses of each proposed approach ;
a Forest plot is a simple way to represent and visualise the results of a benefit-risk assessment; and
the process for developing a standard, simple (not simplistic) approach to benefit-risk assessment should build on a qualitative approach, eventually developing into a quantitative framework.
The CIRS is to hold a follow-up workshop in June 2011 in Washington, DC, entitled Visualising Benefit-Risk: The key to developing a framework that informs stakeholder perspective and clarity of decision making.
1. Refining the Benefit-Risk Framework for the Assessment of Medicines: Valuing and Weighting Benefit and Risk Parameters, CIRS Workshop, 17-18 June 2010, Washington, DC
2. Drug regulation 2011: the developments gathering pace, Regulatory Affairs Pharma, 20 December 2010
3. Mussen F, Salek S and Walker SR, eds, Benefit-Risk Appraisal of Medicines: A Systematic Approach to Decision Making. John Wiley: Surrey, UK, 2009
Stuart Walker is the founder of the Centre for Innovation in Regulatory Science in London, UK.James Neil McAuslane is the scientific director of the centre and Lawrence Liberti is its executive director.
A summary of the Centre for Innovation in Regulatory Science's workshop was published previously (see: S Walker, L Liberti, N McAuslane, Refining the Benefit–Risk Framework for the Assessment of Medicines: Valuing and Weighting Benefit and Risk Parameters, Clin Pharm Ther, 2011:89(2):179-182).