Evaluation of quantitative signal detection in EudraVigilance for orphan drugs: possible risk of false negatives

Abstract

Different strategies have been studied to allow a better characterization of the safety profile of orphan drugs soon after their approval. At the end of the development phases only few data are available because of the small number of subjects exposed to an orphan medicine for the treatment of rare or ultra-rare conditions. As a consequence, the evaluation of the safety profile is limited at the time of the first approval. In the post-marketing period, all available sources should be combined for a better understanding of the safety of orphan drugs. These sources, include outputs from large databases such as the European Medicines Agency’s EudraVigilance database. Analyses of data from this source are required to be performed by marketing authorization holders (MAHs) as part of their signal management activities. In 2018, the Pharmacovigilance Risk Assessment Committee (PRAC) assessed 114 confirmed signals, 79% of which included data from EudraVigilance. MAHs have access to statistical calculations for drug–event combinations (DECs) from EudraVigilance, provided in the form of measures of disproportionality of ratios of the observed proportion of spontaneous cases for a DEC in relation to the proportion of cases that would be expected if no association existed between the drug and the event. However, such statistical summaries for orphan drugs could be misleading because of the very limited safety data available for orphan drugs (under-reporting together with low numbers of exposed patients). In addition, the applied statistical methodology in most instances is constrained by different confounding factors such as indications of specific medicines and the wide spectrum of medical conditions/diseases of patients from whom reporting of disproportionality ratios are derived (i.e. proportions of DECs for orphan drugs (ODECs) from a small patient population suffering the rare disease and the proportion of DECs in the rest of the population represented in the whole database who have been treated with other medicines for a wide range of indications, and prescribed to treat completely different medical conditions). As expected, these statistical calculations produced not only signals of disproportionate reporting (SDRs) that are false positives, but also not sensitive enough to detect certain SDRs, thus resulting in false negatives. In the context of rare/ultra-rare life-threatening diseases where new molecules have been made available on the market on the basis of their proven efficacy, but with only limited safety data at the time of approval, false negatives could be a special concern since unlikely converted in positives or becoming positives with notable delay. Subgroup analyses (using a limited dataset comprising ADRs within specific individual case safety reports (ICSRs), sorted by indication/disease relevant to the drug of interest could, at least in part, possibly reduce some of the weaknesses resulting from the abovementioned confounding factors. On the other hand it could also cause the loss of some identification of SDRs that would be captured if no database restrictions had been undertaken. Therefore, data subgroup analysis should not be selected as a preferred approach to quantitative signal detection for orphan drugs but rather evaluated as complementary possibly to confirm negatives or to further characterize detected SDRs. Some examples of false negatives originating from quantitative signal detection in EudraVigilance applied to orphan drugs are discussed in this article.

Keywords

disproportionality analysis EudraVigilance orphan drugs pharmacovigilance rare condition rare diseases rare disorder signal detection

Introduction

At present, there is no universally accepted definition of rare disease. In the USA, rare diseases are defined as those affecting less than 200,000 individuals (prevalence <650/million population). In Europe, a disease is defined as rare when it affects 5 persons in 10,000 (prevalence: 500/million). A systematic search for definitions related to rare disease from organizations in 32 international jurisdictions resulted in 296 definitions from 1109 organizations published by Richter and colleagues.¹ The average prevalence threshold across organizations within individual jurisdictions ranged from 5 to 76 cases/100,000 people. The global average prevalence from the definitions used in the different jurisdictions would be 40 cases/100,000 people. Ultra-rare diseases are defined as those affecting one patient per 50,000 people (about 20 patients in 1 million). The limited size of a study population for an orphan indication (sometimes only a few dozen or 100 patients) makes the safety profile of an orphan drug only partially defined at the time of approval.^2,3 In general, rare adverse drug reactions (ADRs) are difficult to detect for all drugs but for orphan drugs it is very likely that they remain undetected during the development of orphan drugs and possibly also for long periods following their approval. It is therefore extremely important that the elements and procedures used for safety monitoring for all drugs, but in particular for orphan drugs for rare diseases, are carefully evaluated and based on relevant information available about the drugs from all sources, including chemistry, manufacturing and controls, nonclinical toxicology information and any other data from previous experience in humans. If available, safety profiles of other approved drugs of the same class for other indications should also be used.⁴

Big databases of suspected adverse drug reactions (sADRs) represent an important source of data for signal detection activities. Different statistical disproportionality methods have been evaluated to facilitate the detection of data suggesting potential signals from large databases.^5–12 There is no method of disproportionality calculation preferred to others.^13–17 EudraVigilance is the system for managing and analysing information on suspected adverse reactions to medicines that have been authorized or being studied in clinical trials in the European Economic Area (EEA). The European Medicines Agency (EMA) operates the system on behalf of the European Union (EU) medicines regulatory network. EudraVigilance supports the electronic exchange of ICSRs between EMA, national competent authorities (NCAs), MAHs and sponsors of clinical trials in the EEA, early detection and evaluation of possible safety signals and better product information for medicines authorized in the EEA.

In EudraVigilance different ADR terminologies are standardized by use of the Medical Dictionary for Regulatory Activities (MedDRA). This is a dictionary designed to be used in the registration, documentation and safety monitoring of products during and after the marketing authorization process. Developed by the International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH; multidisciplinary topic M1), MedDRA contains highly specific, standardized medical terminology. The structure of MedDRA is logical. There are five levels in the MedDRA hierarchy, organized from the very specific to the very general. At the most specific level, called ‘lowest level terms’ (LLTs), there are more than 70,000 terms, which reflect how information is communicated. Thus, these LLTs reflects how an observation might be reported in clinical practice. In the next level there are MedDRA ‘preferred terms’ (PTs). These are distinct descriptors (single medical concept) for a symptom, sign, disease diagnosis, therapeutic indication, investigation, surgical or medical procedure, as well as medical, social, or family history characteristics. Each LLT is linked to only one PT. Each PT has at least one LLT (itself). Related PTs are grouped together in ‘high-level terms’ (HLTs) based upon anatomy, pathology, physiology, aetiology, or function. HLTs are, in turn, linked to ‘high-level group terms’ (HLGTs). Finally, HLGTs are grouped into ‘system organ classes’ (SOCs) which are groupings by aetiology (e.g. infections and infestations), anatomical site (e.g. gastrointestinal disorders), or purpose (e.g. surgical and medical procedures). In addition, there is a SOC that includes issues pertaining to product and one containing social circumstances. The use of this drug dictionary is vital to ensure consistency. MedDRA is governed by a maintenance and support service organization (MSSO), which maintains supporting documentation up to date with each release of the dictionary and includes MedDRA training materials. This standardization eliminates the need for conversion from one terminology to another, thus preventing distortion of data. It guarantees consistency through the different stages of development, which facilitates effective cross-referencing. MedDRA is used by regulatory authorities, pharmaceutical companies and clinical research organizations worldwide, and its use is a regulatory requirement of MAHs and marketing authorization applicants in the European Union (EU).

The disproportionality method used in EudraVigilance¹⁸ is the reporting odds ratio (ROR): proportion of cases for a drug–reaction/event combination (DEC) in relation to the proportion of cases that would be expected if no association existed between the drug and the reaction/event. The usefulness of this statistical method for signal detection is based on the hypothesis that when a product causes an event, the number of observed reports for the DEC will tend to exceed the number based on chance alone. The calculation of the ROR is based on a two-by-two contingency table^19–23 (please refer to Table 1).

Table 1.

Two-by-two contingency table used to calculate the reporting odds ratio (ROR) in EudraVigilance.

	Event	Not event
Medicinal product	a⁽¹⁾	b⁽²⁾
No product	c⁽³⁾	d⁽⁴⁾

$R O R = \frac{a / b}{c / d} .$

(1)

Number of individual cases with the suspected medicinal product and the event of interest.

(2)

Number of individual cases with the suspected medicinal product but with no event of interest.

(3)

Number of individual cases with all other suspected medicinal products in EudraVigilance and the event of interest.

(4)

Number of individual cases with all other suspected medicinal products in EudraVigilance and without the event of interest.

A set of rules, known as a signal detection algorithm (SDA), based on the observed value of the disproportionality statistic and, usually, also on other variables (e.g. number of cases reported), is applied in the EudraVigilance Data Analysis System (EVDAS) to indicate when a given DEC should be highlighted as a signal of disproportionate reporting (SDR). SDRs could potentially indicate the presence of a new ADR for the product or a known ADR (i.e. ADR listed in the concerned labelling) reported more than expected in the database. Scientific and medical judgment have to be exercised to determine whether further analyses/evaluations of data is necessary to confirm or refute an SDR as a signal. SDRs are included in electronic reaction monitoring reports (eRMR) made available to MAHs. The eRMR for a particular drug substance is an output containing aggregated data and line listings of suspect ADR cases. Further details of the individual cases per DEC can be generated by MAH via EVDAS. EVDAS users are also able to retrieve ICSR forms accessible through the line listing or directly through the EVDAS interface. The ICSR form provides different levels of access depending on the product’ ownership of the MAHs querying the database and their reported role in the safety report, for example, suspect, concomitant, or interacting.

A SDR is identified when the following conditions are met: (1) ROR lower bound of the 95% confidence interval greater than one in at least one of the geographical regions Europe, North America, Japan, Asia and rest of the world; (2) the presence in EudraVigilance of at least three or five cases including the event of interest with the active substance contained in the medicinal product under signal detection evaluation (threshold set to three cases if the active substance is in the additional monitoring list as per REG 726/2004 EC article 23 and as also defined in GVP Module IX or for paediatric population); (3) the event is included in the important medical event (IME) and/or designated medical event (DME) list as published on the EMA website.^19–23

On the basis of the 2018 Annual Report on EudraVigilance for the European Parliament, the Council and the Commission the database currently holds over 14.5 million individual case safety reports (ICSRs) referring to over 8.3 million cases and is one of the largest pharmacovigilance databases in the world. It has undergone significant development in recent years. This has delivered enhanced functionalities allowing for a better support of pharma-covigilance activities and the protection of public health. Disproportionality methods used in EudraVigilance have demonstrated the ability to detect about 50% of ADRs as compared with other currently used methods of signal detection.²⁴

In 2018, the EMA’s signal management team reviewed in detail 2204 potential signals, that is, drug–event pairs from screening of the EudraVigilance database (78.7%), medical literature (17.8%), or information received from regulatory authorities or other sources.

In the context of a mandated pilot phase of use of eRMRs from EudraVigilance for signal detection by MAHs, which started in February 2018,²⁵ six validated signals had been notified to EMA by the end of 2018.

In 2018, the Pharmacovigilance Risk Assessment Committee (PRAC) of the EMA prioritized and assessed 114 confirmed signals (a 39% increase compared with 2017); 79% included data from EudraVigilance. Fifty of the assessed signals (44%) resulted in a recommendation for an update of the product information for patients and healthcare professionals, thus providing updated guidance on the safe and effective use of the medicines. In six of these cases, the PRAC also recommended a dear healthcare professional communication (DHPC) to highlight new important safety information to prescribers, and in two cases, also to update the risk management plan (RMP). One additional signal led to the update of the RMP to fully characterize and investigate the concern, in the absence of a product information update. In 24 cases (21%) continuing with routine safety monitoring of the medicine was considered sufficient. The evaluation of 39 signals (34%) was ongoing at the time of the annual report including 22 via a follow-up signal procedure and 17 in periodic safety update reports (PSURs)/periodic safety update single assessments (PSUSAs).²⁶

A list of all signals prioritized and assessed by the PRAC in 2018, some of which concerned orphan drugs, has been included in the attachment V to the 2018 annual report on EudraVigilance for the European Parliament, the Council and the Commission.²⁶

Spontaneous reports are submitted to Eudra-Vigilance only when a patient experiences an adverse event that may have been caused by a medicine. The screening of the EudraVigilance data does not provide information about the number of patients taking the medicine who do not experience an adverse event or the number of patients experiencing the same clinical event without taking a medicine, simply because such cases without an adverse event are not reportable to EudraVigilance. As consequence the relevant associations measured at population level (e.g. relative risks or odds) for clinical events for patients taking a medicine compared with those not taking the medicine cannot be calculated. Instead, signal detection relies on disproportionality measures that are relative proportions conditional on reporting to the database.²³ Interpretation of the statistical analyses using data from reporting systems such as EudraVigilance are necessarily accompanied by medical and clinical judgment and consideration of what is known about the product, the disease, the class, the identified and potential risks, exposure, etc.

The ‘crude’ ROR (cROR) applied to orphan drugs, calculated from the ratios of proportions of orphan drug–event combinations (ODECs) and proportion of DECs in the rest of the population present in the whole database treated with other medicines is affected by different limitations. A cROR would result from the comparison of a proportion of a DEC occurring in the small population suffering a rare disease (in most instances life-threatening condition) and the proportion of the same event combined to all the other drugs in the database, administered for a wide range of indications (please refer to Figure 1).

Figure 1.

Crude reporting odd ratio for orphan drugs from EudraVigilance.

The small patient population and the associated small number of cases increases the width of the ROR confidence interval and as result, SDRs may not be identified only because the cROR lower bound of the 95% confidence interval would not be greater than 1 in at least 1 of the 5 geographical regions (but with the other 2 criteria of the SDR fulfilled). Some examples of this effect, which may be false negatives, that is, not determined as SDRs in eRMRs available from EVDAS have been identified for six different orphan drugs and are discussed in this article.

Nevertheless, it is important to note that for the same medicinal products, there were some ODECs that resulted in SDRs matching ADRs already labelled in the concerned summary of product characteristics (SmPC), but these are not discussed in this article because the purpose of this study is to identify whether false negatives might occur.

Material and methods

Between 25 March 2019 and 26 March 2019, eRMRs ‘fixed reference period type’ with no filter applied to the MedDRA hierarchy (i.e. radio button ‘none’ selected for the field ‘Reactions from the MedDRA hierarchy to filter the report results’) have been downloaded from EudraVigilance for six of orphan drugs reported in the additional monitoring list as per REG 726/2004 EC article 23 (eRMR time run on 25/03/2019: cabozantinib, daratumumab and panobinostat; eRMR time run on 26/03/2019: pomalidomide, ponatinib and venetoclax).

For each active substance the concerned eRMR was filtered as follows: (a) the field ‘IME/DME’ only to include the values ‘IME’ or ‘IME/DME’; (b) the field ‘SDR All’ only to include the value ‘No’; (c) the fields ‘ROR (-) Europe’, ‘ROR (-) N America’, ‘ROR (-) Japan’, ‘ROR (-) Asia’ and ‘ROR (-) Rest of the world’ only to include values lower than ‘1’; (d) the threshold was also fixed to include for each region only ODECs including adverse event terms reported at least three times to EudraVigilance. This resulted in a list of terms that were terms included in the IME/DME lists, were a SDR not detected based on the ROR lower bound of the 95% confidence interval, were a ROR from all regions which included values lower than 1 and were terms for which there had been at least 3 case reports reported to EudraVigilance.

For each active substance, the reaction MedDRA preferred terms (PTs) included in the eRMR resulting from the filter described above have been checked against the list of ADRs as included in section 4.8 of the concerned SmPC to determine the presence of possible false negatives in the eRMR (please refer to Figure 2).

Figure 2.

Flow diagram showing the methodology followed to obtain ADRs resulting as false negatives.

Results

The results from this study are provided in Table 2. Overall, a total of 41 MedDRA PTs meeting the criteria of important medical events, each reported at least three times for at least one of the regions (‘ROR (-) Europe’, ‘ROR (-) N America’, ‘ROR (-) Japan’, ‘ROR (-) Asia’, ‘ROR (-) Rest’) and with cROR (i.e. obtained for each region) lower than 1 with the lower bound of the 95% confidence interval have been identified.

Table 2.

MedDRA PTs with cROR lower than 1 with the lower bound of the 95% confidence interval from eRMR obtained from EudraVigilance mapped against the SmPC.

Orphan drug (Active substance)	MedDRA PT	IME/DME (Yes/No)	Tot spont	ROR (–) tot spont Europe⁽⁴⁾	ROR (–) tot spont N America⁽⁵⁾	ROR (–) tot spont Japan⁽⁶⁾	ROR (–) tot spont Asia⁽⁷⁾	ROR (–) tot spont Rest⁽⁸⁾	SDR all	SmPC – ADR frequency ⁽³⁾
Orphan drug (Active substance)	MedDRA PT	IME/DME (Yes/No)	Tot spont	ROR (–) tot spont Europe⁽⁴⁾	ROR (–) tot spont N America⁽⁵⁾	ROR (–) tot spont Japan⁽⁶⁾	ROR (–) tot spont Asia⁽⁷⁾	ROR (–) tot spont Rest⁽⁸⁾	SDR all	VC/C	U	R	VR	NK
CABOZANTINIB ⁽¹⁾	Pancytopenia	Yes	3	0.42	0.00	0.00	0.00	0.00	No
	Neutropenia	Yes	7	0.31	0.25	0.00	0.00	0.00	No	Ѵ
	Myocardial infarction	Yes	4	0.06	0.18	0.00	0.00	0.00	No					Ѵ
	Cardiac failure	Yes	5	0.77	0.00	0.00	0.00	0.00	No
	Cerebrovascular accident	Yes	9	0.06	0.72	0.00	0.00	0.00	No					Ѵ
	Loss of consciousness	Yes	5	0.00	0.56	0.00	0.00	0.00	No
	Seizure	Yes	4	0.03	0.14	0.00	0.00	0.00	No
	Urinary retention	Yes	3	0.76	0.00	0.00	0.00	0.00	No
	Thrombosis	Yes	6	0.13	0.79	0.00	0.00	0.00	No
	Haemorrhage	Yes	8	0.63	0.48	0.00	0.00	0.00	No	Ѵ
DARATUMUMAB ⁽¹⁾	Anaphylactic reaction	Yes	16	0.55	0.55	0.00	0.00	0.00	No			Ѵ
	Cellulitis	Yes	5	0.10	0.18	0.00	0.00	0.25	No
	Peritonitis	Yes	6	0.12	0.13	0.00	0.00	2.28	No
	Loss of consciousness	Yes	18	0.58	0.09	0.62	0.00	0.00	No
	Acute kidney injury	Yes	26	0.96	0.44	0.00	0.10	0.00	No
PANOBINOSTAT ⁽¹⁾	Renal failure	Yes	6	0.81	0.09	0.94	0.00	0.00	No	Ѵ
POMALIDOMIDE ⁽²⁾	Arrhythmia	Yes	36	0.42	0.47	0.23	0.26	0.00	No
	Bradycardia	Yes	15	0.17	0.11	0.03	0.00	0.20	No
	Cardiac arrest	Yes	44	0.27	0.23	0.48	0.13	0.15	No
	Myocardial infarction	Yes	129	0.00	0.44	0.00	0.00	0.00	No	Ѵ
	Deafness unilateral	Yes	9	0.14	0.63	0.48	0.00	0.00	No
	Sudden death	Yes	13	0.00	0.48	0.05	0.11	0.00	No
	Loss of consciousness	Yes	96	0.74	0.47	0.43	0.21	0.00	No
	Respiratory failure	Yes	58	0.57	0.04	0.60	0.00	0.00	No
PONATINIB ⁽²⁾	Gastrointestinal haemorrhage	Yes	6	0.21	0.16	0.00	0.00	0.00	No
	Neuropathy peripheral	Yes	11	0.46	0.97	0.00	0.75	0.00	No	Ѵ
	Renal failure	Yes	16	0.84	0.68	0.00	0.00	0.00	No
	Pulmonary oedema	Yes	7	0.37	0.70	0.00	0.71	0.00	No
	Pulmonary embolism	Yes	9	0.09	0.64	0.00	0.65	0.00	No	Ѵ
VENETOCLAX ⁽²⁾	Arrhythmia	Yes	5	0.14	0.52	0.00	0.00	0.00	No
	Cardiac arrest	Yes	9	0.59	0.31	0.00	0.00	0.00	No
	Cardiac failure	Yes	12	0.84	0.19	0.00	0.00	0.89	No
	Gastrointestinal haemorrhage	Yes	6	0.09	0.14	0.00	0.00	0.24	No
	Lower respiratory tract infection	Yes	7	0.87	0.00	0.00	0.00	0.70	No
	Hyperkalaemia	Yes	8	0.79	0.89	0.00	0.00	0.00	No	Ѵ
	Cerebrovascular accident	Yes	8	0.36	0.15	0.00	0.00	0.00	No
	Coma	Yes	5	0.25	0.27	0.00	0.00	0.00	No
	Loss of consciousness	Yes	7	0.02	0.30	0.00	0.16	0.00	No
	Acute kidney injury	Yes	19	0.70	0.43	0.00	0.00	0.21	No
	Pulmonary oedema	Yes	3	0.00	0.41	0.00	0.00	0.00	No
	Pulmonary embolism	Yes	7	0.07	0.32	0.00	0.00	0.00	No

(1)

eRMRs, fixed reference period, time run on 25 March 2019.

(2)

eRMR, fixed reference period, time run on 26 March 2019.

(3)

Adverse drug reactions (ADRs) reported as: VC/C, very common (⩾1/10) or common (⩾1/100 to <1/10): U, uncommon (⩾1/1000 to <1/100); R, rare (⩾1/10,000 to <1/1000); VR, very rare (<1/10,000); NK, not known (cannot be estimated from the available data) in the Summary of Product Characteristics (SmPC).

(4)

ROR for the region ‘Europe’.

(5)

ROR for the region ‘North America’.

(6)

ROR for the region ‘Japan’.

(7)

ROR for the region ‘Asia’.

(8)

ROR for the region ‘Rest of the World’.

DME, designated medical event; eRMR, electronic reaction monitoring report; IME, important medical event; MedDRA, Medical Dictionary for Regulatory Activity; PT, preferred term; ROR, reporting odds ratio; SDR, signal of disproportionate reporting.

Seven of the 41 MedDRA PTs, matched ADRs noted as ‘very common’ (frequency ⩾1/10) or ‘common’ (frequency ⩾1/100 to <1/10) in the concerned SmPC as summarized as follows:

- neutropenia (cabozantinib: cROR Europe = 0.31; cROR N America = 0.25);

- haemorrhage (cabozantinib: cROR Europe = 0.63; cROR N America = 0.48);

- renal failure (panobinostat: cROR Europe = 0.81; cROR N America = 0.09; cROR Japan = 0.94);

- myocardial infarction (pomalidomide: cROR N America = 0.44);

- neuropathy peripheral (ponatinib: cROR Europe = 0.46; cROR N America = 0.97; cROR Asia = 0.75);

- pulmonary embolism (ponatinib: cROR Europe = 0.09; cROR N America = 0.64; cROR Asia = 0.65);

- hyperkalaemia (venetoclax: cROR Europe = 0.79; cROR N America = 0.89).

Of the 41 MedDRA PTs, one matched with a ‘rare’ (frequency ⩾1/10,000 to <1/1000) ADR in the concerned SmPC: anaphylactic reaction (daratumumab: cROR Europe = 0.55; cROR N America = 0.55).

Two of the 41 MedDRA PTs, matched ADRs with frequency ‘not known’ (frequency cannot be estimated from the available data) in the concerned SmPC summarized as follows:

- myocardial infarction (cabozantinib: cROR Europe = 0.06; cROR N America = 0.18);

- cerebrovascular accident (cabozantinib: cROR Europe = 0.06; cROR N America = 0.72).

For 31 of the 41 MedDRA PTs, an exact match has not been found in the related approved SmPCs.

Discussion

This study investigated some of the limitations of eRMRs obtained from data in EudraVigilance for orphan drugs. As expected, the results from this study have confirmed some weaknesses of the abovementioned statistical calculations, which were not sensitive enough to detect certain ADRs, including some ADRs that are already listed as common or very common complications in the approved labelling of orphan drugs. The study had a specific focus on those SDRs not highlighted as potential SDRs by EVDAS as consequence of cRORs with the lower bound of the 95% confidence interval lower than 1. This may be explained by the small population suffering the rare disease and the very limited safety data available for orphan drugs in EudraVigilance. It is plausible that the under-reporting (e.g. low number of cases for orphan drugs; nonreported ADRs) plays a significant role in causing false negatives because this results in wider confidence intervals for the cROR.

Moreover, in most instances the calculation of cRORs applied to orphan drugs is influenced by different factors such as the indication of the drug, the wide spectrum of medical conditions/diseases of patients from whom reporting of ratios are derived (e.g. ratios of proportions of ODECs in the small patient population suffering the rare disease and proportion of DECs in the rest of the population treated with other medicines for a wide range of indications prescribed for completely different pathologies represented in the whole database). These factors may result in false negatives.

For all six different orphan drugs selected there were ADRs listed in the concerned SmPCs for which a positive SDR was not reported and, therefore, these represent false negatives.

This study did not investigate the generalizability of the same findings to other orphan drugs. It is possible that by applying the same methodology to other substances, which are orphan drugs, different results could be obtained. It should be also noted that this study evaluated eRMRs downloaded from EVDAS only once for each analysed orphan drug. It has not been investigated whether, for the same six orphan drugs used for the analysis, eRMRs downloaded in subsequent periods would provide similar results. This study did not examine further the relevance of the identified ODECs that did not match ADRs listed in the corresponding SmPCs and not defined as SDRs (i.e. whether they could be false negatives/false positives/true negatives/true positives, etc.).

Complementary analyses could be performed to evaluate whether sensitivity of the methodology could possibly be increased by replacing the cROR with a ROR obtained from a dataset filtered by relevant medical condition (disease/indication) of the drug of interest using the MAH EVDAS version. As example of an inappropriate comparison would be for, a SDR for an adverse event of haemorrhage occurring in the in the context of a haemato-oncological rare and life-threatening condition for which an orphan drug has been administered, compared with the SDR for the same event in patients who have been administered an over-the-counter medicinal product for the treatment of a nonserious condition (e.g. treatment with acetyl salicylic acid or with ibuprofen for headache). By filtering the whole EudraVigilance database by medical condition or common pathology ratios of proportions of ODECs in the small patient population suffering the rare disease would be compared with proportions of DECs in patients suffering similar medical conditions (e.g. for a drug used for haematological–oncological disease, sorting the dataset for comparison by haemato-oncological/oncological diseases). However, it is recognized that these details might not always be available in EudraVigilance. It is likewise important to consider that restrictions of the database to accommodate subgroup analysis may introduce other biases potentially causing the under detection of signals that would be detected if no filters to the whole dataset were introduced. As example, for a drug used to treat cancer the restriction of the whole dataset to cancer drugs could result in background disease-related events and ADRs of therapies becoming too similar for drug of interest and the comparator drugs thus resulting in the under-detection of signals. The fundamental basis of use of disproportionality, whereby the variety of drugs and indications form the basis of production of an expected reporting ratio, could break down and even frequently reported events (with an underlying causal link to the drug) might become difficult to detect.

Hauben et al.²⁷ investigated whether some restrictions of data in the Food and Drug Administration Adverse Event Reporting System (FAERS) database could improve oncology drug signal detection performance. In this study positive controls (PCs) were drug–medical concept (DMC) pairs selected from safety information not included in the product’s first label but subsequently added as label changes. These medical concepts (MCs) were then mapped to MedDRA PTs used in FAERS to code adverse events. Negative controls (NCs) were MCs with circumscribed PTs not included in the corresponding US package insert (USPI). The shrinkage-adjusted observed-to-expected (O/E) reporting frequencies for the drug–PT pairs was calculated. An adjudication framework to calculate performance at the MC level was formulated. Performance metrics (sensitivity, specificity, positive and negative predictive value [PPV, NPV], signal/noise [S/N], F and Matthews correlation coefficient [MCC]) were calculated for each analysis and compared. Restriction of the analyses only to oncology drugs improved the S/N ratio, removing proportionately more noise than signal, but with significant credible signal loss. The authors point out that, background restrictions would be expected to decrease the sensitivity while increasing the specificity of observed events or SDRs. Analyses based on an unrestricted database should be the preferred approach for exploratory signal detection due to higher sensitivity. Conducting further analyses based on a restricted database may enable the researcher to further refine and assess SDRs detected during exploratory signal detection due to higher specificity.

Following an approach similar to that described by Hauben and colleagues,²⁷ it would be interesting to investigate whether by applying a Standardized MedDRA Query (SMQ)-based analysis to signal detection for orphan drugs on data from EudraVigilance false negatives would be reduced by grouping different PTs with similar medical concepts. The report on the EVDAS pilot phase for MAHs is expected to produce further clarifications and recommendations on the use of EVDAS MAH version.

It is recognized that these subanalyses described above could be quite difficult to implement, necessitating significant experience and be quite burdensome from a resource perspective. Medical judgment of results of quantitative statistical analyses from databases should, wherever possible, be complemented by qualitative evaluations of ICSRs (for orphan drugs paying attention also at those ADRs not marked as SDRs in eRMRs) by experienced personnel in an attempt to identify potential false negatives.

Footnotes

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Conflict of interest statement

The authors declare that there is no conflict of interest.

ORCID iD

Marco Sardella

References

Richter

Nestler-Parr

Babela

, et al. Rare disease terminology and definitions. A systematic global review: report of the ISPOR rare disease special interest group. Value Health 2015; 18: 906–914.

Price

What can big data offer the pharmacovigilance of orphan drugs?

Clin Ther 2016; 38: 2533–2545.

Sardella

Belcher

Pharmacovigilance of medicines for rare and ultrarare diseases. Ther Adv Drug Saf 2018; 9: 631–638.

Food and Drug Administration. Draft guidance: rare diseases: common issues in drug development. Food and Drug Administration, https://www.fda.gov/ (2015).

Finney

DJ.

Systemic signalling of adverse reactions to drugs. Methods Inf Med 1974; 13: 1–10.

Bate

Lindquist

Edwards

, et al. A Bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharmacol 1998; 54: 315–321.

Bate

Evans

SJ.

Quantitative signal detection using spontaneous ADR reporting. Pharmacoepidemiol Drug Saf 2009; 18: 427–436.

Council for International Organizations of Medical Sciences and Working Group VIII. Practical aspects of signal detection in pharmacovigilance: report of CIOMs Working Group VIII. Geneva: Council for International Organizations of Medical Sciences, 2010.

Evans

Waller

Davis

Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Saf 2001; 10: 483–486.

10.

Szarfman

Machado

O’Neill

RT.

Use of screening algorithms and computer systems to efficiently signal higher-than expected combinations of drugs and events in the US FDA’s spontaneous reports database. Drug Saf 2002; 25: 381–392.

11.

Du Mouchel

Smith

Beasley

, et al. Association of asthma therapy and Churg-Strauss syndrome: an analysis of postmarketing surveillance. Clin Ther 2004; 26: 1092–1104.

12.

Almenoff

Tonning

Gould

, et al. Perspectives on the use of data mining in pharmaco-vigilance. Drug Saf 2005; 28: 981–1007.

13.

van Puijenbroek

Bate

Leufkens

, et al. A comparison of measures of disproportionality for signal detection in spontaneous reporting systems for adverse drug reactions. Pharmacoepidemiol Drug Saf 2002; 11: 3–10.

14.

Hochberg

Hauben

Pearson

, et al. An evaluation of three signal-detection algorithms using a highly inclusive reference event database. Drug Saf 2009; 32: 509–525.

15.

Grundmark

Holmberg

Garmo

, et al. Reducing the noise in signal detection of adverse drug reactions by standardizing the background: a pilot study on analyses of proportional reporting ratios-by-therapeutic area. Eur J Clin Pharmacol 2014; 70: 627–635.

16.

Slattery

Alvarez

Hidalgo

Choosing thresholds for statistical signal detection with the proportional reporting ratio. Drug Saf 2013; 36: 687–692.

17.

Candore

Juhlin

Manlik

, et al. Comparison of statistical signal detection methods within and across spontaneous reporting databases. Drug Saf 2015; 38: 577–587.

18.

European Medicines Agency. Eudravigilance, European database of suspected adverse drug reactions, https://www.ema.europa.eu/en/human-regulatory/research-development/pharmacovigilance/eudravigilance

19.

European Medicines Agency. Guideline on good pharmacovigilance practices (GVP) module IX – signal management (Rev 1), https://www.ema.europa.eu/en/documents/scientific-guideline/guideline-good-pharmacovigilance-practices-gvp-module-ix-signal-management-rev-1_en.pdf (2017).

20.

European Medicines Agency. EV-M5b - EVDAS training for marketing authorisation holders, https://www.ema.europa.eu/en/documents/presentation/presentation-evdas-training-marketing-authorisation-holders-training-module-ev-m5b_en.pdf

21.

European Medicines Agency. Guideline on good pharmacovigilance practices (GVP) module X – additional monitoring, https://www.ema.europa.eu/en/documents/scientific-guideline/draft-guideline-good-pharmacovigilance-practices-module-x-additional-monitoring_en.pdf (2012)

22.

European Medicines Agency. Guideline on the use of statistical signal detection methods in the EudraVigilance data analysis system, http://www.ema.europa.eu/docs/en_GB/document_library/Regulatory_and_procedural_guideline/2009/11/WC500011434.pdf (2017)

23.

European Medicines Agency. Screening for adverse reactions in EudraVigilance. https://www.ema.europa.eu/en/documents/other/screening-adverse-reactions-eudravigilance_en.pdf (2016)

24.

Alvarez

Hidalgo

Maignen

, et al. Validation of statistical signal detection procedures in EudraVigilance postauthorization data: a retrospective evaluation of the potential for earlier signalling. Drug Saf 2010; 33: 475–487.

25.

European Medicines Agency. Update on the pilot of signal detection in EudraVigilance by marketing authorisation holders, https://www.ema.europa.eu/en/documents/other/update-pilot-signal-detection-eudravigilance-marketing-authorisation-holders_en.pdf (2018).

26.

European Medicines Agency. 2018 annual report on EudraVigilance for the European parliament, the council and the commission. Reporting period: 1 January to 31 December 2018. Inspections, Human Medicines Pharmacovigilance and Committees Division, https://www.ema.europa.eu/en/documents/report/2018-annual-report-eudravigilance-european-parliament-council-commission-reporting-period-1-january_en.pdf (2019).

27.

Hauben

Hung

Wood

, et al. The impact of database restriction on pharmacovigilance signal detection of selected cancer therapies. Ther Adv Drug Saf 2017; 8: 145–156.