Hypothesis-free signal detection in healthcare databases: finding its value for pharmacovigilance

Knowledge about a medicine evolves during its lifecycle, as evidence about the product is generated during the pre-clinical, clinical, and post-marketing phases. Pharmacovigilance systems are designed to identify emerging ‘signals’ using multiple data sources, particularly as use of the products expands post-approval in healthcare systems and as treatment pathways evolve. Signals may arise from one or multiple sources and may suggest a potentially new causal association or new aspect of a known risk of a medicine. ¹ Although companies and regulators use multiple data sources to detect emerging signals, spontaneous reports continue to be the cornerstone of signal detection for most marketed drugs and vaccines, as they have been since the 1960s.^2,3 For example, a total of 19 out of 21 drugs withdrawn in France between 1998 and 2004 were reported to be based on spontaneous individual case safety reports (ICSRs). Among these product withdrawals, 12 were solely based on spontaneous reports, 6 were in combination with results from observational studies, and the remaining two were based on randomized controlled trial (RCT) evidence and animal studies, respectively. ⁴ Another study showed that 8 out of 11 product withdrawals between 1999 and 2001 in the UK and US were based on early information from spontaneous reports, for another product solely RCTs were identified as the recorded evidence and for the other two products evidence used to support their withdrawal could not be found in any of the identified documents. ⁵ Spontaneous reports are particularly useful for identifying rare and idiosyncratic safety concerns and acute events.^6,7 Some literature suggests spontaneous reports are most useful for newly marketed products, specifically within 3 years after launch,^8–10 whereas other studies indicated they are also useful for older products.^8,9 A large body of literature has documented the well-known limitations of spontaneous reporting systems (SRS), including underreporting, stimulated reporting, and other reporting biases, the lack of a population denominator to calculate rates, and frequently poor information in terms of both quality and quantity.^3,6

As paper-based reporting systems transitioned to electronic systems and regulatory requirements for reporting have increased, the number of reports received over time by pharmaceutical companies and regulators have risen significantly and further compounded these limitations. Quantitative methods, or measures of disproportionality, were introduced around the turn of the millennium ¹¹ with the goal of making the large numbers of reports more manageable and informative^3,12 by taking advantage of the data volume to assist in identifying new safety signals. ¹³ These methods provide a systematic approach to filter through drug–event pairs and focus the clinical review of drug–event pairs most likely to represent emerging ‘signals of suspected causality’ as CIOMS VIII terms it.^3,14 However, as the scenarios where reports are required by regulators for collection have also expanded (e.g. reports from patient support marketing programs, reports of events occurring during time of drug exposure without suspected causality), there has not been a corresponding improvement in the time between approval and detection of new signals and the duration remained relatively long. ¹⁵ Fukazawa et al. ¹⁵ showed that, using the FDA AERS database, the median duration between approval and detection of signals was 9 years, between approval and label change 10 years. The same study also showed that FDA AERS was less effective for older drugs, that is, more than 5 years in the market. This is not surprising as there are proportionally fewer reports of older drugs and the established weaknesses of spontaneous reporting make such reports less likely to impact the well-established benefit–risk profiles of such medicines. This research as well as our experience at pharmaceutical companies where hundreds of thousands to millions of reports are processed annually, suggests that the increased volume of reports has had a limited impact on the approved product label or assessments of the overall benefit–risk of a medicine.

Until recently, there were no widespread alternative data sources for large scale routine monitoring of the safety profile of products post-launch in the real world setting. There is growing recognition that electronic healthcare databases (for example, insurance claims, electronic medical records, and registry databases), together referred to as real-world data (RWD), ¹⁶ may offer an alternative to SRS. These longitudinal healthcare databases are a rich source of information about the benefits and risks of medical products, as they are longitudinal with anonymized and routinely collected information on millions of patients over years. Given the characteristics of these data sources compared with spontaneous reports databases, for example, duration and completeness of follow up, ease of estimating the population at risk, better ascertainment of exposure and outcomes, and more relevant comparison groups, over time they have the potential to enhance or even replace current signal detection activities within pharmacovigilance systems.

We focus herein on challenges and opportunities to what we term hypothesis-free signal detection in RWD: searching for potential signals across multiple drug/vaccine–adverse event pairs simultaneously and irrespective of whether there is an index of suspicion. ¹⁷ Many of the issues we raise will also apply to the more focused activity of surveillance on predefined outcomes in RWD, an area where there is more scientific activity.^18,19 We do not discuss spontaneous reporting signal detection methods at any more length given how accepted it is, other than to point out how these inform our perspectives on healthcare databases signal detection. Similarly, we consider other existing data streams such as query log (search engine) data ²⁰ and social media^21,22 out of scope, although we recognize these are sometimes categorized as RWD.

RWD in pharmacovigilance: not only signal evaluation

Healthcare databases have been used for epidemiological studies for decades in the evaluation of a priori defined hypotheses (signal evaluation) and to further investigate potential safety signals identified from other sources (signal refinement).^23,24

However, despite the ubiquity of real-world databases for years, their use for hypothesis-free signal detection has been limited. Why is this, given the well-known limitations of SRS and the potential benefits of using real-world databases for signal detection? One of the reasons is that until recently access to such data was costly and limited given the proprietary nature of many databases; thus, access was primarily driven by the need to answer specific research questions or assess specific safety concerns, that is, an a priori hypothesis. Timeliness of the data has also been an issue with substantial delays from the date of an encounter to when it is available in an analytic database. These delays have limited the usefulness of these data for signal detection that relies on timely data and rapid analyses. In addition, data quality and the nonrandom variability on which data are captured or missing across databases makes rapid assessment of multiple outcome pairs across databases extremely difficult, at least currently. All of the above, until recently, made signal detection in RWD far from straightforward operationally and fraught with methodologic pitfalls.

However, with increased access to data through more affordable subscription models, faster computing and data refresh times, common data models (CDMs), ²⁵ and similar enabled software solutions for rapid analytics, the conditions for testing signal detection in RWD has never been better. Compared with spontaneous reports, real-world databases (both electronic medical record databases and insurance claims databases) have several characteristics, some of which are potentially favorable, to enhance signal detection efforts, ¹ including the following.

1. The ability to provide accurate and informative risk estimates, for example, incidence rates, relative risks or incidence rate ratios, as a result of being able to quantify the population at risk and outcomes at baseline and follow-up intervals.

2. Data capture on patients before and after exposure, often for many years. This is in contrast to the cross-sectional nature of most spontaneous reports.

3. Information on comparators, which permits matching (design) or statistical adjustment using epidemiologic study approaches.

4. Reliable data capture for those variables systematically collected for billing or clinical management purposes, even though systems were not developed for research purposes. This is in contrast to the voluntary nature of spontaneous reports.

5. There is no suspicion of causality in coded (structured) data. This provides the ability to detect signals statistically that may never have resulted in a voluntary report, but also presents a potential weakness when compared to spontaneous reports, particularly those with compelling causal rationales (e.g. dechallenge/rechallenge).

6. RWD sources are heterogeneous in their data collection standards and methods, for example, the type of healthcare data captured, stored, and accessible.

7. RWD sources increasingly exist internationally but they are not as pervasive in coverage as spontaneous report systems and until recently mostly covered populations in North America and Europe.

Despite these potential advantages, to date, and to the best of the authors’ knowledge, there is limited routine use of RWD to systematically detect signals potentially associated with medical products. A few public–private initiatives have evaluated the value, feasibility, and utility of observational databases to identify safety issues using various methods, including projects such as the Observational Medical Outcomes Partnership (OMOP), ²⁶ the Innovative Medicines Initiative’s (IMI) PROTECT, ²⁷ Exploring and Understanding Adverse Drug Reactions (EU-ADR), ²⁸ and the Asian Pharmacoepidemiology Network (AsPEN). ²⁹ Each of these projects has contributed to our understanding of the potential of using RWD for signal detection. With the shift in pharmacovigilance toward a more RWD centric approach, we note that companies and regulators are actively testing signal detection methods in RWD. Further work is needed, however, before implementing these activities routinely. The relative performance of methods applied to RWD must be clarified and, importantly, so must the relative value of signals from RWD versus spontaneous reports in terms of signal type, timeliness, and likelihood to yield information that informs patients and physicians through the approved product label or overall benefit–risk assessment.

Comparing methods for signal detection in RWD

With the emergence of ‘Big Data’ and other data science concepts there is widespread attention on further enabling appropriate harvesting of data for secondary use in general.^30,31

The optimal study design and analyses should always be selected based on the appropriateness of the research question (e.g. fit for purpose). Nevertheless, with signal detection, one cannot design bespoke analyses for individual drug–event pairs, given the general objective at hand. That said, is it possible to employ a similar analytic approach for groups of drug–event pairs with specific characteristics? The need for generic approaches, coupled with the complexity and heterogeneity of RWD sources, makes the effective application of signal detection to such data challenging. The solution to the appropriate analytical method for signal detection in RWD may not be a ‘one size fits all’; rather, multiple methods may need to be used in parallel for different drug–adverse event combination groupings.

There have been several methods proposed and tested for signal detection, including the cohort study with propensity score matching, high-dimensional propensity score matching (HDPS), self-controlled case series study (SCCS), tree-based scan statistic, self-controlled cohort analysis with the use of temporal pattern discovery, and Bayesian approaches.^32–42 Existing methods have been applied ‘as is’ or modified slightly for use in RWD signal detection; for example, traditional epidemiologic focused surveillance adapted for hypothesis-free use ⁴¹ and prescription symmetry analysis ⁴³ ; methods historically more associated with disease epidemiology have been implemented, ³⁵ informatics approaches,^44,45 SRS-based methods, ⁴⁶ and approaches novel for this application ⁴² ; as have a combination of the above approaches, as well as ‘advanced analytics’ such as Q-methodology ⁴⁷ and other machine learning and artificial intelligence approaches.^44,48,49 Use of these methods will improve over time as experience with each approach accrues. For example, propensity score matching for signal detection might theoretically be improved by the use of calendar-specific propensity scores, use of multiple comparison groups, and data visualization to better understand characteristics of the study populations, although method testing would be essential to determine whether such changes would translate into meaningful routine performance improvement.

There have been only a limited number of studies^{26,48,50–52} examining the performance (such as sensitivity and specificity) of methods, and only some studies conducted comparative evaluation of more than one method.^35,38 No method has emerged as superior across the range of different exposure and outcomes (with varying covariates of focus) from which emerging signals need to be detected. Research from OMOP, although not explicitly focused on signal detection, identified the self-controlled approach as demonstrating the highest performance characteristics as compared with other study designs, when looking indiscriminately at a large number of different exposure and outcomes with no attempt a priori to optimize the method for a specific drug–outcome pair. ²⁶ The self-controlled study design has several strengths, including the elimination of confounding factors and selection bias, because a person serves as their own control, as well as no need to select a comparator product. This design is best used to study acute or transient outcomes that occur shortly after exposure and not appropriate for fatal outcomes. It seems clear that for some types of exposure and outcomes other approaches for example cohort based, will be needed. There is only one, recent review ⁴⁹ evaluating methods for signal detection in RWD. However the conclusions on relative method performance are necessarily limited as the authors themselves emphasize, owing to the dearth of articles specifically comparing methodologies.

Challenges to implementing signal detection in RWD

We have identified three broad challenges that must be addressed prior to widespread implementing of RWD signal detection into the signal management system: establish RWD is a credible source of signals; demonstrate that tools for RWD signal detection are scalable and rapid; and characterize the best conditions and means to integrate these signals into the overall pharmacovigilance system efficiently and in a way demonstrated to enhance value. In Figure 1, the key questions and actions proposed are described, ordered in a manner that is most likely to lead to the adoption of RWD for signal detection.

Figure 1.

Implementing signal detection in RWD: the necessary steps.

There are additional considerations, not identified in Figure 1, which are also important for operationalizing signal detection in RWD as part of a signal management system. These include the following.

Signal evaluation in the same database(s) as signal detection? Some contend ⁵³ that it is critically important that signal detection, or hypothesis generation, should always be conducted independently using different data sources from those used for signal evaluation, or hypothesis testing epidemiology studies. With a finite number of databases at the field’s disposal (albeit increasing) this may be impractical – particularly given overlapping studies from different groups and the necessity of also conducting feasibility work before initiating studies. Walker proposed ⁵⁴ that if different methods are applied then the same data can in practice be used for two separate analyses (i.e. near-independence between the two) and that this type of data reuse may contribute to better understanding a hypothesis and is distinct from re-using data with the same methods. ⁵⁵ An alternative, given the significant power one often has access to for a given drug-AE pair, might be that, for a given analysis, a database would be partitioned, one part to be used for signal detection and the other for hypothesis testing studies. ⁵⁶ More work is needed to assess how either approach could be used in practice.

Concordance across multiple data streams

Criteria for weighting of outputs from analyses across different RWD data types, as well as other data sources in the pharmacovigilance system, is critical. Complex decision making heuristics for taking into account contradictory evidence across data sets will need to be addressed. In practice, results may vary across different data sources; in other words, Signals of Disproportionate Recording (SDRs) in some data sources may not be SDRs in other data sources or the magnitude of the associations may be different and approaches to integrate results from different data sources, for example with weighting, need be evaluated. The complexity will come in part from the necessity to weigh outputs from the different databases differently, as exposure and outcome varies.

Repeated analyses

One needs to consider the repeated iterative nature of interrogations of different data sources for different drugs and at different points in time. As the amount of data in a database increases over time, signal detections for a single drug in a single database will need to be repeated over time as new signals may be detected and the magnitude of the old signals may change overtime (stronger associations). For this reason, various issues with repeated testing will need to be addressed. For example, if a signal is defined based on p-value, should it be adjusted? One school of thought suggests that no adjustment is needed as this is not a clinical trial. Others advocate some kind of statistical adjustments. Another issue with repeated analysis is how to combine data (for the same signal) across databases and at different points in time. One solution is to apply meta-analyses approaches to creating a single composite measure based on different point estimates and variances.

Complementary nature of RWD

It is important to note that no system is perfect and, even if adequate approaches to the above issues are found, RWD may not be ready to replace SRS, at least not at this point in time, and both sources are complementary to each other. Despite limitations mentioned above, SRS may for example well continue to be very useful, or even better than RWD despite it becoming ever more ubiquitous, for detection of rare events in the early years after launch. RWD may be superior to SRS for events that require a longer induction period or those that are acute, especially when the self-control case series is used. Some evaluations suggest such differential performance of the data streams^28,57 but more evaluations are needed to see how the two sources complement each other, when and in what situations RWD is superior to SRS and should be used instead of SRS, and vice versa. There may even be situations in the future where neither RWD nor SRS is most suitable. For example, given the large amount of data and the high speed of the accumulation of data in social media, this data source may at some point in the future be found to be even faster to detect certain specific events for newly launched drugs than both RWD and SRS.

The development and implementation of signal detection capabilities within a signal management system needs to take a practical approach that evolves through knowledge and experience before adoption as a best practice to the field of drug safety. The initial approach should be pragmatic, have realistic expectations that should not cause an over-reaction to the process or to the results generated from signal detection efforts. The evolution of a robust process will take time to develop, learn, and demonstrate its value to the overall signal management system. It is important to have individuals with experience, in pharmacovigilance, drug safety surveillance, but also epidemiology and healthcare databases, involved in the process to apply clinical judgement in the interpretation of potential signals identified via signal detection capabilities. There should also be areas of opportunities to collaborate and share learnings from these efforts with key stakeholders. The ultimate goal should be the creation of a rapid, scalable, and value-added approach to effectively supplement and enhance current signal detection capabilities.

Conclusion

The use of real-world healthcare databases for signal detection offers an opportunity to enhance current signal detection activities. It could first be augmented to the current system and eventually replace outmoded approaches that fail to add value in a modern pharmacovigilance system. Experience to date suggests a distinct role for SRS remains when compared with RWD for signal detection. ⁵¹ We expect this will evolve in the ensuing years. Signal detection in RWD is likely to make some spontaneous reporting processes redundant and to elucidate those data sources, such as patient support programs, ²¹ where active monitoring overall adds little to the approved product label, benefit–risk knowledge or patient safety. The circumstances when signal detection in RWD will complement or replace current activities require further investigation.

A large-scale approach, that is, most drugs for multiple events, where the best method for each outcome pair is applied is unlikely. For example, self-controlled designs may be expected to be beneficial for signal detection in acute outcomes where exposure is short term and accurately measured, so accurate time between exposure and outcomes can be identified and the within patient control brings benefit in terms of confounding control. ³³ When considering cancer outcomes, one would need to consider cohort or case control studies as explored in recent research. ⁵⁸ Rather selecting the best approach across a broad range of outcomes for a drug class or outcomes across drug classes is likely to be a necessity. As a result, outputs with such a large-scale approach will contain more false-positives than formal hypothesis testing studies. This means signals emerging from RWD must be triaged into routine signal management processes and undergo the essential clinical review phase and comparison to other surveillance outputs. ⁵¹

To reach the goal of a pharmacovigilance system enhanced by signal detection in RWD, explicit guidance on how signal detection is best performed in RWD and indeed other emerging data streams, ²⁷ must be a priority for collaborative work across stakeholders. We envision a forum with participation from across industry, regulatory, and other government agencies and patient/healthcare provider advocacy groups to collaborate on answering questions about methodologies and the best ways to operationalize signal detection in RWD in a manner that is efficient and avoids redundancy. The FDA have recently held a public meeting focused on signal detection methods in the Sentinel Network. ⁵⁹ This is an important start but will require transparent, extensive foundational research and collaboration with stakeholders about practical approaches to be successful. Developing a harmonized approach and pooling resources is also critical and likely the best means to resolve the current uncertainty about the contribution signal detection in RWD will make to safety surveillance strategies. Eventually guidance documents, similar to those completed for SRS in an FDA-industry collaborative white paper ² and in the CIOMS VIII, ¹ reflecting an harmonized regulatory view globally will be needed. Signal detection in RWD holds promise for enhancing overall signal detection capability of pharmacovigilance systems. To harness this promise, in the coming years, it will be critical that we all work together to identify the best way to advance signal detection in RWD to ensure patient safety and protect public health.