Abstract
Background:
Mobile applications (apps) have become a ubiquitous source of clinical decision support. We sought to ascertain the feasibility of using an app platform to obtain a crowdsourced measure of adverse drug reaction reporting rates associated with sugammadex administration and compare it with traditionally-derived estimates.
Methods:
Using the widely-distributed anesthesia calculator app, ‘Anesthesiologist’, we surveyed anesthesia providers regarding their experience with adverse drug reactions associated with sugammadex administration.
Results:
Data were analyzed from 2770 participants in 119 countries responding between March 2016 and May 2017, who were estimated to have administered between 1.6–2.9 million doses (588–1040 administrations per participant). A low and high-end reporting rate of adverse events was estimated based on respondents’ reported frequency and duration of sugammadex use. The estimated reporting rate of anaphylaxis due to sugammadex was 0.0055–0.098%, similar in range to previously published estimates.
Conclusions:
Use of an in-app survey facilitated a global assessment of anesthesia providers and could have useful applications in monitoring adverse events and estimating their rates. Further work is needed to validate this approach for other medications and clinical domains.
Introduction
Postmarketing surveillance of adverse drug reactions (ADRs) are a critical element in the assessment of drug safety, as clinical trials performed prior to approval may not capture their full range or incidence. 1 Voluntary spontaneous reporting mechanisms are used worldwide in nearly every country in the world; one example is the United States (US) Food and Drug Administration’s (FDA) Adverse Event Reporting System (FAERS). Calculating ADR rates from these sources is complicated by under-reporting bias; it is well established that such calculations are inappropriate under most circumstances given that their main intended purpose is signal detection.2–5 Given the limitations inherent with voluntary reporting systems, there is a clear push to use novel data sources to augment our ability to detect postmarketing ADRs and calculate their rates.6–8 A recently published editorial summarized the myriad approaches being used, including social media and patient-facing mobile health applications (apps) for signal detection. 9
Provider-facing apps are one promising source of ADR data. Apps are widely used by both patients and healthcare workers, with 259,000 apps available across multiple sources as of 2016. 10 Cloud-based techniques are being developed to use apps as platforms for two-way information exchange, providing app-based medical content on the one hand while gathering data from users on the other. 11 For pharmacovigilance in particular, the Innovative Medicines Initiative WEB-RADR project is deploying apps for ADR reporting the European Union, and elsewhere, as well as investigating the value of social media in the identification of drug safety concerns.12,13
We sought to ascertain whether data collected from providers via an app platform could be used to estimate ADR rates. In particular, we were interested in exploring the rate of anaphylaxis and other ADRs associated with sugammadex administration. This drug is of interest because approval for clinical use in the US was delayed for several years primarily over concerns about hypersensitivity reactions and effects on coagulation. 14 A dedicated hypersensitivity trial elucidated the incidence of anaphylaxis as 1 per 299 cases (0.33%), but other sources suggest the rate is lower.15–18 By 2015, approximately 11.5 million dose of sugammadex had been distributed in countries across Europe, Asia, and Central and South America.15,19 Therefore, we queried anesthesia providers worldwide about their experience related to this and other known ADRs. We hypothesized that the crowdsourced estimate would be consistent with existing ADR rate estimates.
Materials and methods
Ethics approval and manuscript preparation
The study was reviewed and approved by the Emory University Institutional Review Board (study number 00082571), including a waiver of written informed consent. Participants anonymously gave electronic consent before participating in any data collection. The US FDA had previously determined that the app used for this study falls into the category of ‘enforcement discretion,’ meaning that, at present, the US FDA does not intend to enforce requirements for this app under the Federal Food, Drug, and Cosmetic Act. 20 This manuscript was prepared in accordance with the STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) checklist for improved reporting of outcomes from observational studies. 21
Data collection
A free app (‘Anesthesiologist’) for Android mobile devices, providing anesthesia references and drug calculation capabilities, was previously developed and deployed.11,22 This app was built as a clinical resource for anesthesia providers and has over 50,000 active users. The Survalytics platform was integrated with the app, enabling cloud-based delivery of survey questions and storage of survey responses and application analytics using an Amazon (Seattle, WA, USA) Web Services (AWS) database. 23 Anonymized information including basic demographics (Online Resource 1, Table A), user location, and app usage data was collected. Further details regarding data collected and the methodology are available in Online Resource 1. An anonymous 10-question survey was deployed via the app; questions relevant to the work presented here are shown in Table 1. Data analyzed in the present work were collected between December 2015 and May 2017, with initial deployment of the sugammadex survey occurring in March 2016. After opting in to study participation, users were presented with survey questions upon opening the app and had the option of clicking a ‘Not now/answer later’ button to bypass the survey, in which case the most recent question was presented again with the next app activation. Only users of the app who reported access to sugammadex (Table 1, Q-02, ‘Yes’) were presented with additional questions in the survey. Only data from physicians and providers in anesthesia-related professions were used in the present analysis; data from other subjects (e.g. nurses, paramedics, medical student, pharmacists, and respiratory therapists) was discarded.
Survey for collection of information about sugammadex.
Statistical methods
Raw data from the AWS DynamoDB table were downloaded and analyzed in R v3.4.2 (R Core Team, Vienna, Austria).24,25 For calculation of confidence intervals of event rates, the Clopper–Pearson method was applied using the ‘PropCI’ R package. 26 In some cases where confidence intervals were not supplied with the primary source, we performed this calculation using numbers from that source to back-calculate confidence intervals.
The significance of differences between demographic characteristics were calculated using the Wilcoxon rank-sum test. Respondents submitting the default numbers for the two questions assessing their usage (for Q-03: 5 years and for Q-04: 25 administrations per week) were assumed to represent failure to change the default position (‘straight-line answering’) rather than true responses and were excluded from the calculation. We calculated a raw ADR prevalence based on the reported number of events divided by the total number of respondents. We estimated ADR reporting rate by dividing the total number of ADRs reported by the total number of doses of sugammadex administered. Because we did not collect information from providers on the number of times that they had observed each ADR, nor did we collect information about the number of weeks per year worked, the values used for the numerator and denominator were calculated using two scenarios: (1) a high ADR/low-use scenario, in which respondents were assumed to have observed the reported ADR 10 times and to work 26 weeks per year, and (2) a low ADR/high use scenario, in which respondents were assumed to have observed each reported ADR once and to work 46 weeks per year. For the high ADR scenario, we chose to multiply the reported number of events by 10 in order to account for under-reporting bias and assume an observed event rate far in excess of the likely actual rate that an individual provider would have observed each event. Additional details are provided in Online Resource 1. We also assessed for differences in the likelihood of reporting an ADR based on the participant’s total number of administered doses, excluding participants calculated to have administered zero doses (e.g. either zero years or zero uses per week reported). In Table 4, we report the median and interquartile range of the low-use scenario number of administered doses, and significance was calculated using the Wilcoxon rank-sum test.
Results
Participant characteristics
A total of 12,450 anesthesia providers in 187 countries responded to the screening question regarding access to sugammadex and whether it was relevant to their clinical practice (Table 2; response rate = 12.5% based on an estimated maximum of 100,000 active app users). A total of 6489 (52%) anesthesia providers in 167 countries indicated they had access to sugammadex and were presented with the remainder of the survey. As users could cease answering questions at any time, there was attrition in the total number of responses recorded for each question (i.e. ‘respondent fatigue’, see Discussion); this attrition is presented in Figure 1. Characteristics associated with respondent fatigue have been analyzed separately. 27 Demographic information, as available based on data collected in the basic survey (Online Resource 1, Table A), is presented in the first column of Table 3.
Participant self-report of availability and relevance of sugammadex to their practice.
CRNA, certified registered nurse anesthetist; PA, physician’s assistant.
Number of participants providing answers to each question in the sugammadex survey. Respondent fatigue led to attrition in the total number of responses collected for Q-03 versus Q-10.
Demographic characteristics of study participants.
ADR, adverse drug reaction; CRNA, certified registered nurse anesthetist; PA, physician’s assistant.
Reporting rate of ADRs related to sugammadex administration
In addition to respondent fatigue, we also observed an unexpectedly large number of responses where the sliding-bar response mechanism used to collect the answer to Q-03 and Q-04 were not changed from their default positions (‘straight-line’ answering). For Q-03, 2329 (35.9%) of participants entered the default response of 5 years. For Q-04, 1966 (30.3%) of participants entered the default response of 25 administrations per week. Together, this represented data from 2679 (41.3%) respondents. After excluding responses from these respondents, data from 2770 participants in 119 countries were available for analysis. The demographic characteristics of this subset of participants is presented in the second column of Table 3. As seen in Online Resource 1, Figure A, the majority of providers participating in the survey used sugammadex less than five times per week and had used it for less than 3 years.
Collectively, participants had administered an estimated 1.6–2.9 million doses of sugammadex, or 588–1040 administrations per participant. As shown in Table 4, participants who had administered a greater number of doses of sugammadex were more likely to have reported an ADR in this survey. The ranges for the estimated reporting rate of sugammadex-related ADRs are reported in Table 4. Anaphylaxis rates were estimated to be between 1:1000 and 1:20,000 (0.005–0.098%). Bradycardia and incomplete reversal of neuromuscular blockade (NMB) were the most common events reported, with a high-end estimated rate of approximately 1:500. Hypersensitivity and bronchospasm were less common than anaphylaxis. A small number of users reported off-label use of sugammadex to treat anaphylaxis caused by rocuronium or vecuronium. Overall, a cumulative 22.7% of participants reported observing an ADR associated with sugammadex administration.
Crowdsourced estimate of the prevalence and reporting rate of adverse events associated with sugammadex administration. Low and high-end estimates of the rate were based on assumptions about how much providers worked and how many events each report (N) represented. Participants who used sugammadex more were more likely to report an ADR.
ADR, adverse drug reaction.
Discussion
Using an app-based approach, we surveyed more than 6000 anesthesia providers in 167 countries. Responses from nearly 3000 users were used and accounted for an estimated 1.6–2.9 million doses of sugammadex administered. We found that a small minority of providers (5.7%) reported that they had encountered an anaphylactic reaction. This crowdsourced data yielded an estimated rate of 1:1000–1:20,000 for anaphylaxis to sugammadex. The most commonly reported ADRs were bradycardia and incomplete reversal of NMB.
Several approaches have previously been taken to characterize the rate of anaphylaxis to sugammadex. Those rate estimates are summarized in Figure 2. Postmarketing surveillance relies on voluntary reporting by healthcare professionals and consumers, which is subject to under-reporting and is therefore unreliable for the calculation of ADR frequency. 3 Even so, the postmarketing data for sugammadex has been used to support its safety profile. A presentation on sugammadex to the US FDA Advisory Committee notes that 273 cases of anaphylaxis were reported in postmarketing surveillance for 11.5 million doses distributed. 15 Using these data, a rate of 24 per 100,000 (0.024%) was suggested using the following assumptions: 95% of distributed vials were used and only 10% of cases were reported. 15 There were two other estimated anaphylaxis rates that were cited in that presentation: (1) an upper limit on the rate of 0.1% based on pooled phase I–III data with no cases of anaphylaxis in N = 3519 patients, and (2) a rate of 0.33% based on the clinical trial requested by the US FDA, in which one patient had anaphylaxis out of N = 299 patients (notably, the Clopper–Pearson 95% confidence interval for that study puts the rate between 0.0085% and 1.9%, an interval too wide to be of value in clinical decision-making). A recently published study by Miyazaki and colleagues reported an estimated anaphylaxis rate of 0.059%, with a 95% confidence interval of 0.032–0.1%. 17 At our institution, we have observed just one case of anaphylaxis in 5321 administrations, yielding a confidence interval of 0.004–0.135%. Comparison of these rates demonstrates that the app-based approach is consistent with rates estimated by these other methods. That participants who had used sugammadex more were more likely to have observed an ADR lends further strength to our findings; given that our survey would have only captured prevalent events, it would be expected that increasing usage of the drug would increase the likelihood of observing an ADR related to that drug.
Comparison of the rates of anaphylaxis to sugammadex from previously published sources and from our estimate. As the estimated incidences span several orders of magnitude, the figure is plotted on a log scale. Where one was calculated, the raw incidence is marked and the Clopper–Pearson range is added. For the pooled phase I–III findings, the event rate was 0, so the Clopper–Pearson range provides the upper limit indicated (the lower limit is zero). For the app-based estimate, the mean reporting rate is marked with a star.
Another potential source of internal validation to this study would have been the ability to compare reporting rates for other ADRs. Unfortunately, anaphylaxis is the most well studied of the ADRs associated with sugammadex. The only other ADR for which there is publicly available information is the reporting rate of bradycardias associated with sugammadex that were presented to the US FDA Advisory Panel by the manufacturer. We used this to calculate a Clopper–Pearson confidence interval, and present the comparison in the Online Resource, Figure B. Of note, in the presentation, bradycardias had fewer spontaneously reported events than anaphylaxis. In our survey, we found bradycardia events to be more commonly reported. It could be that prompting led more respondents to recall a specific bradycardia event, whereas this potentially less impactful ADR is under-reported on a spontaneous basis. This is a speculative conclusion that requires further work to truly understand.
A few of our findings merit further discussion. First, 99 providers reported using sugammadex to treat allergic reactions due to rocuronium or vecuronium. This off-label use of sugammadex has been described in a number of case reports but is controversial and cannot be recommended without additional formal investigation. 28 Second, 7% of respondents (N = 202) had observed residual NMB with sugammadex usage. We have previously reported that cost concerns were the dominant force in provider decision-making about sugammadex use. 29 As residual blockade is known to occur when sugammadex is under-dosed, a speculative conclusion regarding this finding is that cost concerns could drive under-dosing and an increased frequency of residual blockade.30,31 Finally, a number of providers reported ‘other’ ADRs not listed in the question. For simplicity of the survey instrument and to encourage maximum thoughtful response rate, we did not follow up on this. However, our survey could easily be modified to follow up with open-ended questions regarding the ADR report, potentially extending the value of this approach to signal detection of rare and otherwise unrecognized ADRs.
Of interest in future investigations will be the rate at which data can be collected using this methodology. This will be highly dependent on the type of app, the frequency with which it is used, the target audience (e.g. for patient care providers), and the total user base. After all screening and filtering, we obtained 2770 responses over a 15 month period, or 184 responses per month. Further work is necessary to understand how generalizable our findings are to other apps in terms of ability to collect data and the speed of data collection. Another area of future investigation will be the capacity for this methodology to assess geographic distributions of ADRs. Our data were too limited to reach any conclusions about geographic variance in ADR distribution due to the possibility of sampling bias (see Online Resource, Table B).
This exploratory approach to estimate the reporting rate of ADRs has several limitations. Primarily, our approach is likely biased in several ways, and the absence of methodologically similar comparator data is an important limitation in this context. The low-end estimate is likely biased towards underestimation. Respondents may not have recalled or recognized a sugammadex associated ADR, resulting an artificially low numerator. As a relatively new drug, sugammadex usage is likely to have been increasing over time. However, collected responses may reflect more recent utilization, not the average historic utilization. This source of recall bias would result in an artificially large denominator.
The high-end estimate, on the other hand, is likely biased towards overestimating the event rate: we severely discounted the time providers are likely to have worked, and even though some respondents may have observed events more than once, the model uses a generous reported to actual multiplier of 10. Therefore, the true rate of anaphylaxis most likely lies somewhere within our reporting range of 0.005–0.1%.
Another limitation is that the app serving as the platform is primarily used as a reference for pediatric patients, and providers using the app may skew towards pediatric specialists. 11 This is an important consideration as sugammadex may see less use in this patient population due to less frequent NMB agent use. 32 In the US and several other countries, sugammadex has off-label status for patients younger than 17 years old. 33 Withdrawal bias may have influenced our results as there were statistically significant differences between the population of users starting the survey and those providing enough/appropriate data for use in the final calculation of ADR rate.
One source of selection bias may be due to the relatively low survey response rate. Another source of selection bias may be due to exclusive use of the Android platform. Prior reports suggest that there do not appear to be significant personality differences between users of Android versus iOS operating systems. 34 Android devices occupied 82% of the global smartphone market 35 in 2016, but with higher relative penetration of the generally more expensive devices running iOS in high-income countries. 36 Therefore, differences in participant income or other unaccounted-for variables could skew the app user base and introduce confounding.
We observed respondent fatigue in our survey. A term from the survey literature, respondent fatigue describes situations in which respondents fail to complete the survey or provide less than completely thoughtful answers to questions in the survey, particularly with questions placed towards the end of the instrument. 37 Respondent fatigue potentially introduces bias into the survey results in that the missing data may not be occurring at random. An analysis of our findings related to respondent fatigue demonstrated that specific demographic categories were associated with respondent fatigue (measured by the failure to complete a secondary survey). 27 In order to minimize respondent fatigue, we limited the number of questions and restricted ourselves to multiple choice questions and sliding-bar responses.
Conclusion
Even in light of clear limitations, using an app-based approach to assess ADR rates appeared to recapitulate the estimate of a significant ADR obtained by other methods. In particular, several studies (including ours) have now found a rate of anaphylaxis to sugammadex significantly lower than the rate listed in the package insert. Our work demonstrates the potential feasibility of using app-based surveys for postmarketing ADR signal detection and calculation of an ADR reporting rate. If validated, and subject to the limitations discussed, app-based surveys may be a useful novel data stream in the identification of drug administration patterns, monitoring of utilization rates, and detection and characterization of ADRs by tapping into the experience of healthcare professionals regionally and globally.
Key points
The reporting rate of anaphylaxis due to sugammadex, based on the app-based crowdsourced data, was estimated to be not more than 0.1%. This estimate carries substantial limitations in accuracy, but was consistent with previously reported estimates using other, more standard approaches.
App-based approaches may be valuable adjuncts to other traditional and developing sources of information for adverse event signal detection and assessment of reporting rates for adverse events.
Supplemental Material
Supplemental material – Supplement for Crowdsourcing sugammadex adverse event rates using an in-app survey: feasibility assessment from an observational study
Supplemental material, Supplement for Crowdsourcing sugammadex adverse event rates using an in-app survey: feasibility assessment from an observational study by Craig S. Jabaley, Francis A. Wolf, Grant C. Lynde and Vikas N. O’Reilly-Shah in Therapeutic Advances in Drug Safety
Footnotes
Acknowledgements
Craig Jabaley and Francis Wolf contributed equally to this work.
George Easton (Associate Professor of Information Systems & Operations Management, Goizueta Business School, Emory University, Atlanta, GA, USA) developed the methodology for calculating the frequency of app use. Scott Gillespie (Senior Biostatistician, Emory University School of Medicine. Atlanta, GA, USA) provided the R code for mapping provider country of origin income level and provided statistical advice with regards to handling of the calculation of adverse event prevalence and incidence. For critically reading and commenting on the manuscript, we thank: W. Timothy Garvey, Professor and Chair, Department of Nutrition Sciences, University of Alabama, Birmingham, USA and Daniel Green, Epidemiology and Cancer Control, St. Jude Children’s Research Hospital
The study was reviewed and approved by the Emory University Institutional Review Board (study number 00082571), including a waiver of written informed consent. Participants anonymously gave electronic consent before participating in any data collection. The US FDA had previously determined that the app used for this study falls into the category of ‘enforcement discretion,’ meaning that, at present, the US FDA does not intend to enforce requirements for this app under the Federal Food, Drug, and Cosmetic Act
Authors’ note
Craig S. Jabaley and Francis A. Wolf contributed equally to this article.
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. The Emory University Department of Anesthesiology generously supported the time of the authors in producing this work.
Conflict of interest statement
All authors declare: no support from any organization for the submitted work; no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years; no other relationships or activities that could appear to have influenced the submitted work. The app was initially released in 2011 by Vikas O’Reilly-Shah with advertising in the free version and a paid companion app to remove the ads. The app intellectual property was transferred to Emory University in 2015 and advertisements were subsequently removed, and the companion app to remove ads made freely available for legacy users not updating to the ad-free version. Following review by the Emory University Research Conflict of Interest Committee, Vikas O’Reilly-Shah has been released from any conflict of interest management plan or oversight.
Supplementary material
Supplementary material is available for this article online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
