Benefit–risk evaluation: the past,present and future

Abstract

In the last two decades there has been a shift in the approach to evaluating the benefit–risk (BR) profiles of medicinal products from an unstructured, subjective, and inconsistent, to a more structured and objective, process. This article describes that shift from a historical perspective; the past, the present, and the future, and highlights key events that played critical roles in changing the field.

Keywords

benefit–risk methods patient preference quantitative structured weight

Introduction

The past: subjective assessment

In the late 1990s and early 2000s, while there was an increasing pressure from regulatory agencies for pharmaceutical companies to perform BR evaluation more routinely and systematically, there were only a few publications in the literature and a few guidelines from the regulators on how to perform BR analyses. The majority of the publications in the literature were line listings of benefits and risks which might be useful for clinicians but, without taking into account the relative importance of benefits to risks (or preference weight), these could be interpreted differently by different stakeholders, leading to inconsistency in the interpretation.^1–4

In 1998, the need for a more systematic and consistent approach to combining the benefits and risks was first introduced by the Council for International Organizations of Medical Sciences (CIOMS), in the report of CIOMS Working Group IV, Benefit–risk balance for marketed drugs: evaluating safety signals:⁵ ‘If the benefits of the various options under consideration can be assumed to be equal, benefit–risk evaluation can rely on measures of relative risk. Otherwise, in the absence of a readily available and quantitative relationship between benefits and risks, which is commonly the case, evaluation usually comes down to analyses and conclusions that rely on indirect, informal and unavoidably subjective processes.’ CIOMS recognized that the common practice of BR evaluation at that time was subjective in nature and further mentioned that there were no methods to evaluate BR profiles of medicinal products comparing different treatments with relative merits, ‘There are no accepted general methods for deriving a benefit–risk ratio or another composite metric, or for using such measures to compare relative merits of alternative treatments. As ordinarily used, therefore, the BR ratio compares figuratively, but not often quantitatively, the relative magnitudes of benefits and risks.’ The terms ‘semiquantitative’ and ‘quantitative’ in this report referred to the quantification of the benefits and risks, such as severity of events for semiquantitative and cumulative incidence for quantitative measures, and not for the preference weight comparing benefits and risks.⁵

By 1999, there was still no guidance from the regulatory perspective. The European Committee for Proprietary Medicinal Products recommended methods to evaluate risks in the postmarketing settings (such as observational studies) but offered no clear guidance on how to perform BR evaluation, taking into account the preference weight between the benefits and the risks and: ‘. . .[w]henever possible, both benefits and risks should be considered in absolute terms and in comparison to alternative treatments. The degree of risk that may be considered acceptable is dependent on the seriousness of the disease being treated.’⁶ The US Food and Drug Administration (FDA) provided guidance on the benefits and risks but not on BR evaluation.⁷

Two papers that proposed a more structured and quantitative BR evaluation were published in the early 2000s, proposing a few quantitative BR methods taking into account the preference weight of benefits and risks.^8,9 The papers proposed number needed to treat (NNT) and number needed to harm (NNH) to measure the effects for benefits and risks, respectively, and relative value adjusted NNT to not only take into account benefits and risks but also the preference weight of benefits and risks. While the concepts of NNT, NNH, and relative value to adjusted NNT were not new and introduced as early as 1988,^10–14 these two papers by Holden and colleagues^8,9 offered a different perspective and generated a lot of interest for further evaluation of the BR field, followed by key public–private partnerships in this field.

The present: structured and quantitative framework

Starting in the mid 2000s, the BR evaluation field has shifted toward a more structured and quantitative approach. In 2006, following the publication of the paper by Holden and colleagues,⁸ the Benefit–Risk Action Team (BRAT), a collaborative project on BR evaluation sponsored by The Pharmaceutical Research and Manufacturers of America was initiated,¹⁵ followed by several other regulatory [European Medicines Agency (EMA) and FDA] and public–private partnership projects a few years after. Some key projects in the field are highlighted in chronological order here.

The BRAT framework

The BRAT framework is a process to perform BR evaluation in a structured, transparent, and consistent way.¹⁵ The process consists of six steps (define the context, identify outcomes, identify data sources, customize framework, assess outcome importance, and display and interpret key BR metrics);¹⁵ it helps to inform stakeholders to make BR decisions, to communicate the decisions and the rationales for the decisions, and therefore increases the transparency of the whole process. The framework is flexible to incorporate benefits, risks, and preference weight from different perspectives, and standardized to ensure transparency and consistency. It is also simple and easy to understand, which is very important to improve the chance of successful application, especially in an organization with a complicated decision-making structure with numerous conflicting agendas.

Another framework that is comparable with BRAT is PrOACT–URL (problem, objectives, alternatives, consequences, trade-off, uncertainly, risk tolerance, and linked decisions).^16,17 Both BRAT and PrOACT–URL are standardized, yet flexible, frameworks suitable for incorporating outcomes and preference weight. Although BRAT is simpler (with fewer steps) than PrOACT–URL, there is no clear advantage of one over the other. In addition, both frameworks meet the necessary steps for an effective BR evaluation: define the context in which the decision is being made, identify the important relevant information and data regarding benefit and risk, assess the preference weight, make a decision from the information based on expert judgment, and communicate the decision and its rationale.¹⁸ While BRAT and PrOACT–URL allow for structured BR evaluations, neither of them offer quantitative methods to integrate benefits and risks and incorporate preference weight into the BR evaluation.

EMA benefit–risk methodology project

The EMA recognized the need to develop a more structured approach to evaluating the BR profiles of medicinal products to ensure transparency and consistency across different stakeholders and started a 3-year ‘Benefit–Risk Methodology’ project in 2009.¹⁹ Based on the early results from the project, the investigators introduced a two-level approach to performing BR evaluation: first, a qualitative approach, mainly consisting of key effects of the benefits and risks and their uncertainty, and second, recommended for more complex situations, a quantitative approach utilizing quantitative methods to incorporate preference weight. One specific quantitative method, the multicriteria decision analysis (MCDA), was specifically mentioned, although it was recognized that the method had significant challenges for successful implementation that needed to be addressed.¹⁹

In its final report in 2012, Benefit-risk methodology project, work package 4 report: benefit–risk tools and processes, the EMA’s suggestions were more specific:²⁰

(1) PrOACT–URL model for qualitative approach which may be sufficient for most cases, or

(2) MCDA model for quantitative app-roach, following the eight steps consistent with PrOACT–URL, which are suitable for more complex situations.

Besides complicated situations, the MCDA model was also suggested to be used when BR balance was ‘marginal’ and there was no clear BR advantage of one treatment over the others. EMA also considered that the quantitative BR model could play a key role for European regulators to monitor the BR profile of a medicinal product postapproval and to update the model with new data to assess whether the BR profile has changed.²⁰

FDA benefit–risk framework

In 2009, the same year the EMA started its BR methodology project, in response to feedback from FDA stakeholders and recognizing the need to improve the clarity and transparency of FDA’s BR evaluations, the FDA initiated an effort to explore more systematic approaches to BR assessment and communication.¹⁸ In this effort, the FDA concluded that a structured qualitative approach would be more appropriate, as it was flexible to accommodate quantitative analysis and that the use of quantitative analysis was supporting, rather than replacing, judgment. The FDA also considered that quantitative methods might not capture the nuanced assessments and they might obscure subjective expert judgment.¹⁸

The FDA developed a structured BR framework with key decision factors as follow: analysis of condition, current treatment options, benefit, risk, and risk management; each factor with two components: (a) evidence and uncertainties; and (b) conclusions and reasons (Figure 1).¹⁸ While the EMA suggested PrOACT–URL for qualitative, and MCDA for quantitative evaluation in its BR methodology project report, the FDA did not prescribe any methods in its structured BR framework.

Figure 1.

FDA benefit–risk framework.

As part of the fifth authorization of the Prescription Drug User Fee Act (PDUFA V), in 2013, the FDA published the draft implementation plan of the BR framework for the fiscal years 2013–2017. As part of PDUFA VI, the FDA would continue the implementation of the structured BR framework in the fiscal years 2018–2022 and had made several commitments, including holding a meetings to gather stakeholder input and conducting a second evaluation of the implementation of the BR framework starting in 2021.²¹

IMI: PROTECT

Started in 2009, the Pharmacoepidemiological Research on Outcomes of Therapeutics by a European Consortium (PROTECT) was a collaborative European project under the umbrella of the Innovative Medicine Initiative (IMI) with an aim to address limitations of current methods in the field of pharmacoepidemiology and pharmacovigilance. One of its working programs, the BR group was established to develop methods for use in BR assessment, including both the methods and the presentation of the results, with a particular emphasis on graphical methods.²² A total of 47 BR methods were identified, thirteen of which selected for further investigation in the case studies;²³ lessons learned from the PROTECT BR group were summarized and synthesized into the BR roadmap.²⁴

IMI–PROTECT contributed significantly to the BR field especially by providing deeper and more careful evaluations of a wide range of BR methods that could be categorized as follows:²⁴

(1) Frameworks: BRAT and PrOACT–URL;

(2) Quantitative methods: MCDA and stochastic multicriteria acceptability analysis;

(3) Metrics: NNT, NNH, impact numbers, and BR ratio;

(4) Estimation techniques: probabilistic simulation model, indirect/multiple treatment comparison;

(5) Utility survey technique: discrete choice experiment (DCE).

Results from these evaluations have been publis-hed.^25–33 The IMI – PROTECT BR group work has also been used as a reference in regulatory guidance documents and in their working plan.³⁴

In addition, there were other projects that contributed to the shift of BR evaluation toward a more structured and quantitative approach: the IMI–ADVANCE (Accelerated Development of Vaccine Benefit–Risk) initiative which started in 2013 with an aim to develop and test methods and guidelines for BR evaluations vaccines that are on the market³⁵ and the UMBRA (Unified Methodologies for Benefit–Risk Assessment) initiative, which was started in 2012 by The Center for Innovation in Regulatory Science (CIRS), resulting in the UMBRA framework that has been used by regulatory agencies in Canada, Australia, Switzerland, and Singapore.³⁶

All of these projects and initiatives contributed to a current practice of BR evaluation, not only among pharmaceutical industry but also among regulators in their decision-making process, which was more structured, transparent, and consistent. Most of the analyses were qualitative in nature, but quantitative methods were also available and accepted by regulators for a more complex situation where there was no clear advantage of one treatment over the other. When a quantitative approach was necessary, preference weight was needed from different stakeholders, including patients.

It was not clear how much these frameworks had made an impact among pharmaceutical industry and regulators, but a number of studies using the BRAT framework had been published.^37–40 The BRAT framework was also used in a number of IMI–PROTECT case studies²² and it was consistent with the FDA structured BR approach.¹⁸ Two BR studies using PrOACT–URL have been published^40,41 and it was used in the EMA BR Project.¹⁹ A small number of publications of BR studies using these two frameworks might not fully capture the extent to which they have been implemented in the pharmaceutical industry and regulators. Because there were a large number of pharmaceutical companies as well as regulators in these initiatives, it was likely that the frameworks had been used for internal purposes or as part of their regulatory requirements, including FDA advisory committee meetings. The results were not published probably because of proprietary and confidentiality issues.

Results of BR evaluations using quantitative methods have also been published before.^40,42–44 It was not clear whether the regulators used quantitative BR methods. The FDA did not recommend any specific quantitative method, although the use was encouraged to support the qualitative approach.¹⁸ Although one of the suggestions from the EMA BR project was to use MCDA for a more complex situation,¹⁹ it was not clear whether MCDA had been used in the EMA decision-making process. One of the reasons for the regulators’ reluctance to formally use quantitative BR methods might be to maintain their neutral position and not to favor one method over the other. Another possible reason was that, given the complexity of many of the methods, they might not yet have the capability to implement them.

The future: patient perspective in BR evaluation

The role of patient preferences in the BR evaluation of medicinal products and devices has become increasingly important in the last several years and the patient perspective has become an integral part of the regulatory decision making. Patients’ roles not only include the expression of patients’ preferences to be used in the BR evaluation during regulatory approval or postmarketing evaluations, but also in discovery and development. In the discovery phase, patients’ feedback can help to identify unmet medical needs to inform the design of the target product profile, better understand the disease and acceptability of benefits and risks. In clinical development, patient preferences can inform clinical trial design through identification of patient-relevant outcomes, for example, but can also inform product design validation and provide insights on acceptability of benefits and risks.⁴⁵

The shift toward a focus on patient preference started as early as 2009 when a case study on patient preference in BR evaluation was done by the IMI–PROTECT BR Group.²⁶ In this study, patients with obesity were contacted to participate in the BR evaluation of rimonabant (a CB1 receptor antagonist) with a comparison groups using DCE method.²⁶ Afterwards, there were a number of initiatives on patient preference by the FDA, EMA and a number of public – private partnerships.^45–51

From the regulatory perspective, the shift toward a more patient-centered BR evaluation started in the early 2010s. In 2012, as part of PDUFA V, the FDA held the first ‘Public meeting and request for comments’ on patient-focused drug development⁴⁶ and a total of 24 meetings during the fiscal years 2013–2017 were planned. In 2012, the FDA Center for Devices and Radiological Health (CDRH) published a guidance that included quantitative patient-preference data on the trade-off between benefits and risks as a factor in the regulatory review; this guidance was revised in 2016.⁵² The FDA also published a guidance for ‘Patient preference information: voluntary submission, review in premarket approval applications, humanitarian device exemption applications, and de novo requests, and inclusion in decision summaries and device labeling’ in 2016.⁴⁷ An initiative on patient preference elicitation was announced by the EMA in 2015, although patient representatives have been participating in the EMA Scientific Advisory Group meetings since 2011.⁴⁸ As expected, this trend toward a focus on patient preferences among regulatory agencies impacted the research agenda, not only among regulators but also academia and the pharmaceutical industry.

The first public–private partnership with significant impact on patient preference in the BR evaluation was the Medical Device Innovation Consortium–Patient-Centered Benefit–Risk (MDCI–PCBR) Project in the US, with an objective of improving medical device regulatory science for patient benefit.⁴⁹ The MDIC, founded in 2012, performed the PCBR project in response to the FDA CDRH’s priority to focus on BR assessment as a central component of the medical device approval process. Representatives from the FDA, NIH, industry, nonprofit, and patient organizations participated in the project. The MDIC–PCBR project had two main components: (a) the development of a framework underpinning BR assessment with patient preference; and (b) the development of the catalog of methods that can be used not only to collect patient preference but also to incorporate into the BR analysis. Both the framework and catalog of methods were very useful and have been used by different public–private partnerships, pharmaceutical companies, and other stakeholders to perform patient-centered BR evaluations which have become increasingly important for meeting regulatory requirements.

In Europe, another public–private partnership was formed, focusing on ‘. . .education and training to increase the capacity and capability of patients to understand and contribute to medicine research and development and also improve the availability of objective, reliable, patient-friendly information for the public.’⁵⁰ This project, another one under the umbrella of IMI, the European Patients’ Academy (EUPATI) was a European consortium of representatives from the pharmaceutical industry, academia, not-for-profit, and patient organizations.⁵⁰ While EUPATI did not provide or propose any methods for the collection of patient-preference information or incorporating it in the BR evaluation, it plays an important role in the education and training of patients, critical to the success of a patient-centered BR evaluation.

Started in 2016, also under IMI, the Patient Preferences in Benefit and Risk Assessments during the Treatment Life Cycle (PREFER) project is a 5-year public–private research initiative with representatives from academia, the pharmaceutical industry, and patient organization and health technology assessment bodies, with an aim to strengthen patient-centric decision making throughout the life cycle of medicinal treatments by developing expert and evidence-based recommendations to guide the stakeholders.⁵¹ One of the strengths of the PREFER initiative is that, learning from previous initiatives by the FDA, EMA, IMI PROTECT, MDIC and EUPATI, PREFER’s approach was broader, by incorporating perspectives from a wide set of stakeholders.⁵¹ It will evaluate the stakeholders’ expectations and concerns about the assessment of patient preferences and their use in decision making. Based on this, a systematic and comprehensive review of patient-preference methods will be performed, with input from experts in the field. Another strength is that a number of methods would be further tested and evaluated in a number of clinical case and simulation studies focusing on different decision points in the medical product life cycle.⁵¹ Results from the PREFER initiative will contribute significantly not only to the patient-preference research area but also to BR fields. Results on factors and situations that influence the value of patient-preference studies along the medical product lifecycle have been published.⁴⁵

It is not clear how much impact these initiatives on patient preference have made in the pharmaceutical industry, and regulators in their decision-making processes, but several studies using patient-preference data in BR evaluations have been published before.^38,53–59 FDA CDRH had implemented patient preference in the BR evaluation of medical devices earlier than the Center for Drug Evaluation and Research (CDER). CDRH published the draft guidance for quantitative patient-preference data in 2012 which was then revised in 2016;⁵² CDER published its draft guidance on patient-focused drug development 6 years later in 2018.⁶⁰ With its collaboration with the MDIC, the CDRH performed the first public-partnership project on patient preference;⁴⁹ CDER has not performed any public–private initiative on this topic. CDRH also performed the first study of quantitative patient-preference data to inform its approval decision.⁵⁸ In this study, discrete-choice experiments were used to quantify patient preference in comparing different weight-loss devices among US population with obesity.⁵⁸ There has been no published study on patient preference by CDER.

Final remarks

There has been great improvement in the BR field in the last 2 decades, from a subjective and inconsistent, to a more structured, transparent and consistent approach, with a number of quantitative methods to incorporate preference weight from various stakeholders. While this development is encouraging, there is still more work to be done.

Communication of BR evaluation to patients and other stakeholders is one of the areas that need to be improved. How could BR decision and the rationale behind it be communicated effectively and transparently with an opportunity for a routine dialog between decision makers and stakeholders? When should communication between regulators and stakeholders start in the drug lifecycle? In various public partnerships, less attention has been given to the communication than to other aspects of BR evaluation. Another area is the patient perspective; how, when, and what to elicit regarding the patient perspective and how to incorporate it in the BR evaluation. A number of past and ongoing initiatives will address these issues to a certain degree, but education and training of patients and other stakeholders will take a longer time, and more efforts will be required to enable patients to participate in these discussions. BR evaluations must be meaningful; not only to regulators and pharmaceutical companies but most importantly, to patients.

Footnotes

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

Conflict of interest statement

Juhaeri Juhaeri is an employee of Sanofi, a biopharmaceutical company that manufactures various drugs and biologics. While he did not receive a specific grant for this manuscript, he received salary and restricted shares from Sanofi. In addition, he owns an investment account which, at certain points in time, may include investment in biopharmaceutical business.

ORCID iD

Juhaeri Juhaeri

References

Cocchetto

Nardi

RV.

Benefit-risk assessment of investigational drugs: current methodology, limitations, and alternative approaches. Pharmacotherapy 1986; 6: 286–303.

Aaron

Harbach

Mattano

et al. Risk and benefit evaluation in development of pharmaceutical products. Environ Health Perspect 1993; 101(Suppl. 3): 291–295.

Rawlins

Trading risk for benefit. Risk and consent to risk in medicine. London: Parthenon Publishing, 1989, pp.193–202.

Edwards

Wiholm

Martinez

Concepts in risk-benefit assessment. A simple merit analysis of a medicine?

Drug Saf 1996; 15: 1–7.

CIOMS. Benefit-risk balance for marketed drugs: evaluating safety signals. Geneva: CIOMS Working Group IV, Council for International Organizations of Medical Sciences (CIOMS), 1998, pp. 21–22.

Pharmacovigilance Working Party. Notice to marketing authorisation holders: pharmacovigilance guidelines. London: European Agency for the Evaluation of Medicinal Products, 1999, p. 17.

U.S. Food and Drug Administration. Guidance for Industry Development and Use of Risk Minimization Action Plans. Rockville, MD: U.S. Department of Health and Human Services, Food and Drug Administration, 2004, p. 4.

Holden

Juhaeri

Dai

Benefit-risk analysis: a proposal using quantitative methods. Pharmacoepidemiol Drug Saf 2003; 12: 611–616.

Holden

Juhaeri

Dai

Benefit-risk analysis: examples using quantitative methods. Pharmacoepidemiol Drug Saf 2003; 12: 693–697.

10.

Laupacis

Sackett

Roberts

RS.

An assessment of clinically useful measures of the consequences of treatment. N Engl J Med 1988; 318: 1728–1733.

11.

Cook

Sackett

DL.

The number needed to treat: a clinically useful measure of treatment effect. BMJ 1995; 310: 452–454.

12.

Sackett

Deeks

Altman

DG.

Down with odds ratios!

Evidence-Based Med 1996; 1: 164–166.

13.

Mancini

Schulzer

Reporting risks and benefits of therapy by use of the concepts of unqualified success and unmitigated failure: applications to highly cited trials in cardiovascular medicine. Circulation 1999; 99: 377–383.

14.

Guyatt

Sinclair

Cook

et al. Users’ guides to the medical literature: XVI. How to use a treatment recommendation. Evidence-based medicine working group and the cochrane applicability methods working group. JAMA 1999; 281: 1836–1843.

15.

Coplan

Noel

Levitan

et al. Development of a framework for enhancing the transparency, reproducibility and communication of the benefit-risk balance of medicines. Clin Pharmacol Ther 2011; 89: 312–315.

16.

Hunink

Weinstein

Wittenberg

et al. Decision making in health and medicine: integrating evidence and values. Cambridge: Cambridge University Press, 2014. pp. 5–26.

17.

Hammond

Keeney

Raiffa

Smart choices: a practical guide to making better decisions. Boston, MA: Harvard Business School Press, 1999, pp. 4–14.

18.

Food and Drug Administration. Structured approach to benefit–risk assessment in drug regulatory decision-making: draft PDUFA V implementation plan. Fiscal years 2013–2017. Rockville, MD: U.S. Department of Health and Human Services, 2013.

19.

Zafiropoulos

Phillips

Pignatti

et al. Evaluating benefit–risk: an agency perspective. Regulatory Rapporteur 2012; 9: 5–8.

20.

European Medicines Agency. Benefit-risk methodology project, Work package 4 report: benefit-risk tools and processes. Canary Wharf, London, UK: European Medicine Agency, Human Medicines Development and Evaluation, 2012.

21.

US Food and Drug Administration. Benefit-risk assessment in drug regulatory decision-making 2018: draft PDUFA VI implementation plan (FY 2018-2022), 30 March 2018. Rockville, MD: U.S. Department of Health and Human Services, Food and Drug Administration.

22.

The Pharmacoepidemiological Research on Outcomes of Therapeutics by a European Consortium. WP5: benefit-risk integration and representation, http://www.imi-protect.eu/wp5.shtml (2013 Accessed on 19 July 2019).

23.

Mt-Isa

Hallgreen

Wang

et al. Balancing benefit and risk of medicines: a systematic review and classification of available methodologies. Pharmacoepidemiol Drug Saf 2014; 23: 667–678.

24.

Hughes

Waddingham

Mt-Isa

et al. Recommendations for benefit-risk assessment methodologies and visual representations. Pharmacoepidemiol Drug Saf 2016; 25: 251–262.

25.

Juhaeri

Mt-Isa

Chan

et al. IMI work package 5: report 1:b:i benefit-risk wave 1 case study report: rimonabant. Report No.: 1. London: Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium, 2013. http://www.imi-protect.eu/documents/JuhaerietalBenefitRiskWave1CasestudyreportRimonabantOct2011.pdf (accessed 19 July 2019).

26.

Juhaeri

Amzal

Chan

et al. Benefit-risk wave 2 case study report: rimonabant. In: Benefit-risk integration and representation. Report No.: 2:b:i. London: Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium, 2013. http://www.imi-protect.eu/documents/JuhaerietalBenefitRiskWave2CaseStudyReportRimonabantJan2012.pdf (accessed 19 July 2019).

27.

Micaleff

Callreus

Phillips

et al. Benefit-risk wave 1 case study report: efalizumab. In: Benefit-risk integration and representation. Report No.: 1:b. London: Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium, 2013. http://www.imi-protect.eu/documents/Micaleff_et_al_Benefit_Risk_Wave_Case_study_Report_Efalizumab_Feb_2013.pdf (accessed 19 July 2019).

28.

Nixon

Nguyen

TST

Stoeckert

et al. Benefit-risk wave 1 case study report: natalizumab. In: Benefit-risk integration and representation. Report No.: 1, May 2013. http://www.imi-protect.eu/documents/NixonetalBenefitRiskWave1casestudyreportNatalizumabMay2013.pdf (accessed 19 July 2019).

29.

Nixon

Waddingham

Mt-Isa

et al. Benefit-risk wave 2 case study report: natalizumab. In: Benefit-risk integration and representation. London: Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium, 2013. http://www.imi-protect.eu/documents/NixonetalBenefitRiskWave2CasestudyReportNatalizumabMarch2013.pdf (accessed 19 July 2019).

30.

Phillips

Amzal

Asiimwe

et al. Benefit-risk wave 2 case study report: rosiglitazone. In: Benefit-risk integration and representation. Report No.: 2:b:ii. London: Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium, 2013. https://pdfs.semanticscholar.org/f38b/6780d6cf6bdd69f8793f0db3c882dd891b38.pdf (accessed 19 July 2019).

31.

Quartey

Hallgreen

Chan

et al. Benefit-risk wave 1 case study report: telithromycin. In: Benefit-risk integration and representation. Report No.: 1:b:ii. London: Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium, 2012. http://www.imi-protect.eu/documents/QuarteyetalBenefitRiskWave1CasestudyReportTelithromycinFeb2012.pdf (accessed 19 July 2019).

32.

Mt-Isa

Peters

Phillips

et al. Review of visualisation methods for the representation of benefit-risk assessment of medication: stage 1 of 2. Report No.: 2:i. London: Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium, 2013. http://www.imi-protect.eu/documents/ShahruletalReviewofvisualisationmethodsfortherepresentationofBRassessmentofmedicationStage1F.pdf (Accessed 19 July 2019).

33.

Mt-Isa

Hallgreen

Alex Asiimwe

et al. Review of visualisation methods for the representation of benefit-risk assessment of medication: stage 2 of 2. In: Benefit-risk assessment and representation. Report No.: 2:ii. London: Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium, 2013. http://protectbenefitrisk.eu/documents/ShahruletalReviewofvisualisationmethodsfortherepresentationofBRassessmentofmedicationStage2A.pdf (accessed 19 July 2019).

34.

European Medicines Agency. Committee for medicinal products for human use (CHMP): work plan 2018. 2018. https://www.ema.europa.eu/en/documents/work-programme/committee-medicinal-products-human-use-chmp-work-plan-2018_en.pdf (accessed 19 July 2019).

35.

Innotive Medicines Initiative. Accelerated development of vaccine benefit-risk collaboration in Europe, https://www.imi.europa.eu/projects-results/project-factsheets/advance (2013 accessed 19 July 2019).

36.

The Centre for Innovation in Regulatory Science. Unified methodologies for benefit-risk assessment, http://www.cirsci.org/decision-making-frameworks/umbra-initiative/ (2012 accessed 19 July 2019).

37.

Levitan

Andrews

Gilsenan

et al. Application of the BRAT framework to case studies: observations and insights. Clin Pharmacol Ther 2011; 89: 217–224.

38.

Levitan

Phillips

Walker

Structured approaches to benefit-risk assessment: a case study and the patient perspective. Ther Innov Regul Sci 2014; 48: 564–573.

39.

Hallgreen

Van den Ham

Mt-Isa

et al. Benefit-risk assessment in a post-market setting: a case study integrating real-life experience into benefit-risk methodology. Pharmacoepidemiol Drug Saf 2014; 23: 974–983.

40.

Nixon

Dierig

Mt-Isa

et al. A case study using the PrOACT-URL and BRAT frameworks for structured benefit risk assessment. Biom J 2016; 58: 8–27.

41.

Wen

Zhang

Yang

Two approaches to incorporate clinical data uncertainty into multiple criteria decision analysis for benefit-risk assessment of medicinal products. Value Health 2014; 17: 619–628.

42.

Roldan

Badia

Marcos-Rodriguez

et al. Multi-criteria decision analysis as a decision-support tool for drug evaluation: a pilot study in a pharmacy and therapeutics committee setting. Int J Technol Assess Health Care 2018; 34: 519–526.

43.

Rutten-van Mölken

Leijten

Hoedemakers

et al. Strengthening the evidence-base of integrated care for people with multi-morbidity in Europe using multi-criteria decision analysis (MCDA). BMC Health Serv Res 2018; 18: 576.

44.

Schey

Krabbe

Postma

et al. Multi-criteria decision analysis (MCDA): testing a proposed MCDA framework for orphan drugs. Orphanet J Rare Dis 2017; 12: 10.

45.

Van Overbeeke

Whichello

Janssens

et al. Factors and situations influencing the value of patient preference studies along the medical product lifecycle: a literature review. Drug Discov Today 2019; 24: 57–68.

46.

Food and Drug Administration. Prescription drug user fee act V patient-focused drug development; consultation meetings; request for notification of patient stakeholder intention to participate. Rockville, MD: U.S. Department of Health and Human Services, Food and Drug Administration, 2012.

47.

US Food and Drug Administration. Patient preference information–voluntary submission, review in premarket approval applications, humanitarian device exemption applications, and De Novo requests, and inclusion in decision summaries and device labeling. Silver Spring, MD: U.S. Department of Health and Human Services, Food and Drug Administration, Center for Biologics Evaluation and Research, 2015.

48.

European Medicines Agency. Scientific Advisory Groups (SAG): experience and impact of patient involvement. Human Medicines Evaluation Division, EMA, 2014. https://www.ema.europa.eu/documents/presentation/presentation-scientific-advisory-groups-sag-experience-impact-patient-involvement-francesco-pignatti_en.pdf (accessed 16 May 2019).

49.

The Medical Device Innovation Consortium. Patient centered benefit-risk (PCBR), https://mdic.org/project/patient-centered-benefit-risk-pcbr/ (2012 accessed 19 July 2019).

50.

The European Patients’ Academy. The IMI EUPATI project, 2012. https://www.eupati.eu/ (accessed 19 July 2019).

51.

De Bekker-Grob

Berlin

Levitan

et al. Giving patients’ preferences a voice in medical treatment life cycle: the PREFER public-private project. Patient 2017; 10: 263–266.

52.

US Food and Drug Administration. Guidance for industry and food and drug administration staff: factors to consider when making benefit-risk determinations in medical device premarket approval and de novo classifications. Rockville, MD: U.S. Department of Health and Human Services, Food and Drug Administration, 2016.

53.

Katz

Hauber

Gopal

et al. Physician and patient benefit-risk preferences from two randomized long-acting injectable antipsychotic trials. Patient Prefer Adherence 2016; 10: 2127–2139.

54.

Whalley

Hauber

Crawford

et al. Patients’ priorities for treatment in severe asthma. Value Health 2015; 18: A504.

55.

Okumura

Inoue

Yasaka

et al. Comparing patient and physician risk tolerance for bleeding events associated with anticoagulants in atrial fibrillation-evidence from the United States and Japan. Value Health Reg Issues 2015; 6: 65–72.

56.

Levitan

Markowitz

Mohamed

et al. Patients’ preferences related to benefits, risks, and formulations of schizophrenia treatment. Psychiatr Serv 2015; 66: 719–726.

57.

Yuan

Levitan

Burton

et al. Relative importance of benefits and risks associated with antithrombotic therapies for acute coronary syndrome: patient and physician perspectives. Curr Med Res Opin 2014; 30: 1733–1741.

58.

Gonzalez

Lerner

et al. Incorporating patient-preference evidence into regulatory decision making. Surg Endosc 2015; 29: 2984–2993.

59.

Holmes

EAF

Plumpton

Baker

et al. Patient-focused drug development methods for benefit-risk assessments: a case study using a discrete choice experiment for antiepileptic drugs. Clin Pharmacol Ther 2019; 105: 672–683.

60.

US Food and Drug Administration. Patient-focused drug development: collecting comprehensive and representative input 2018, https://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM610442.pdf (accessed 19 July 2019).