Override rate of drug-drug interaction alerts in clinical decision support systems: A brief systematic review and meta-analysis

Abstract

Primary studies have demonstrated that despite being useful, most of the drug-drug interaction (DDI) alerts generated by clinical decision support systems are overridden by prescribers. To provide more information about this issue, we conducted a systematic review and meta-analysis on the prevalence of DDI alerts generated by CDSS and alert overrides by physicians. The search strategy was implemented by applying the terms and MeSH headings and conducted in the MEDLINE/PubMed, EMBASE, Web of Science, Scopus, LILACS, and Google Scholar databases. Blinded reviewers screened 1873 records and 86 full studies, and 16 articles were included for analysis. The overall prevalence of alert generated by CDSS was 13% (CI95% 5–24%, p-value <0.0001, I^2 = 100%), and the overall prevalence of alert override by physicians was 90% (CI95% 85–95%, p-value <0.0001, I^2 = 100%). This systematic review and meta-analysis presents a high rate of alert overrides, even after CDSS adjustments that significantly reduced the number of alerts. After analyzing the articles included in this review, it was clear that the CDSS alerts physicians about potential DDI should be developed with a focus on the user experience, thus increasing their confidence and satisfaction, which may increase patient clinical safety.

Keywords

clinical decision support system computerized physician order entry drug-drug interactions medication safety systematic review user experience

Introduction

Medication prescription errors are quite prevalent worldwide and an important threat to patient safety. Although the most common results are only mild adverse effects, some cases significantly increase the risk of death.¹ In this context, harmful drug-drug interactions (DDIs), which can occur when the effects of one drug are influenced by the effects of another, are the leading cause of this risk. Research has already proven this and indicated that over half of adverse drug effects are directly related to prescription medication errors.^2–5

To reduce these risks, healthcare systems around the world are developing and implementing electronic health records (EHR) with clinical decision support systems (CDSS) that warn prescribers of potential DDIs (pDDIs), thus protecting patients from adverse drug events (ADE).⁶ Potential DDIs can be predicted from knowledge about the pharmacological properties of the drugs prescribed. There are different databases where healthcare professionals can query about potential drug interactions. In the Drugbank database alone, which is one of the most used drug databases, there are at least 365,984 interactions recorded.⁴

Despite the concern with the information presented by CDSS to medical teams, it is also necessary to consider how these teams use the medical record and report their difficulties. The concept of user experience (UX) deals with how users interact with a product based on their expectations and needs in relation to the use of the product in question since bad experiences may decrease the use of the tool or even the complete discontinuation.⁷ This phenomenon can also occur with CDSS, and in the case of detecting pDDIs, a negative UX can cause low adherence of healthcare professionals to the system’s guidelines.^8–10

Given that it is important to verify whether this issue could impact patients’ clinical safety and further develop this tool by focusing on the UX, which is crucial to improving prescriber adherence. In order to better understand the issues related to these CDSS, we sought to conduct a systematic search to learn from the experiences of other CDSS implemented in other countries’ healthcare systems. Other systematic reviews have focused on prescribing errors that can be avoided by CDSSs,^11–14 although none have focused on pDDI alerts generated by these systems.

Given the above, this study aimed to assess the frequency of CDSS-generated pDDI alerts in EHRs and evaluate adherence to these alerts through the alert override rate of physicians utilizing this tool to prescribe drugs.

Methods (study design)

The methodological procedure was defined by following the Preferred Reporting of Systematic Reviews and Meta-Analysis (PRISMA) guidelines.¹⁵ The review question was formulated according to the PICOS approach (problem, intervention, control, outcomes, and study design): “ How are physicians’ adherence to pDDI alerts in using electronic health records with a clinical decision support system capable of identifying drug-drug interactions in their prescriptions? ” Problem: Physicians’ prescriptions with pDDIs made on EHRs at a hospital and/or primary health care unit. Intervention: The use of a clinical CDSS that can detect and inform DDIs. Comparator: None. Primary outcome: The prevalence of DDI alerts in each setting. Secondary outcome: The prevalence of alert override by physicians. Study: Retrospective (comparative, cross-sectional, case-control, and cohort) and prospective (comparative, cohort) studies reporting the use of a CDSS tool used to identify DDIs at hospitals and/or primary health care units (Appendix A).

Literature search

A systematic literature search was conducted on April 12, 2023, in the electronic databases MEDLINE/PubMed, EMBASE, Web of Science, Scopus, and LILACS, searching for articles published between 2011 and 2023. Only manuscripts published in the English, Spanish, and Portuguese languages were included. We also searched for studies in gray literature using the Google Scholar database. A pilot search was conducted to define the Medical Subject Headings (MeSH) search terms and strategies, which was validated by three experts (CLC, JFH, and RSW). A list of reference articles (indicated by experts) was used to define the search strategy. A MEDLINE/PubMed search strategy was developed using a combination of the following keywords, which were divided into 3 research blocks: ((“Decision Support Systems, Clinical”[Mesh] OR “Decision Support Systems” OR “Information Systems”[Mesh] OR “Clinical Pharmacy Information Systems”[Mesh]) AND (“Drug Interactions”[Mesh] OR “Drug interactions”) NOT (“Herb-Drug Interactions”[Mesh] OR “Food-Drug Interactions”[Mesh])). The search strategy used in MEDLINE/Pubmed was adapted for other databases and is presented in Appendix B. The references retrieved from the searches were organized in EndNote Reference Manager web (2021) and Rayyan QCRI (Qatar Computing Research Institute - Data Analytics, Doha, Qatar) online software.^16,17

Study selection

Independent pairs of reviewers (MF, RAS, DS, and EMD) selected the articles based on the inclusion and exclusion criteria. The inclusion criteria were: physicians’ prescriptions with pDDIs made on EHRs at a hospital and/or primary health care unit that uses CDSS. The reviewers began by reading the titles and abstracts independently while applying the eligibility criteria, followed by full-text reading, also applying the eligibility criteria. A third reviewer (CLC, JFH, and RSW) cross-checked all the retrieved information in both phases. The final selection was always based on the full text of the publication and according to the PICOS approach (Appendix A). We excluded studies that did not satisfy the inclusion criteria and those that met the exclusion criteria (Appendix C), and the remaining full-text articles were used for the study. We also searched for articles in the reference lists of eligible studies.

Data extraction

The articles were randomly distributed to four authors to extract the data independently using the Research Randomizer® software.¹⁸ Data were extracted from the text, figures, or tables and added to a standardized table. In addition, three email attempts were sent to the authors whose studies were included in this review to acquire missing data and provide further clarification on the subject. The extracted data were validated by a pair of reviewers (MF, RAS, DS, and EMD). Any discrepancies were resolved through discussion or consulting a third reviewer. Afterward, the expert group validated the data extracted in the standardized form (Supplementary Material, Table S1). An expert group (CLC, JFH, and RSW) checked the extracted data, and disagreements were solved through discussion. The data were exported to the R software¹⁹ for meta-analysis and quality assessment analysis.

Data synthesis

A random effects model for meta-analysis in the Rstudio software¹⁹ was used for statistical pooling of data. The dichotomous outcomes were reported as prevalence ratios (events/total) according to 95% confidence intervals. Meta-analysis was performed using a minimum of three studies, and the pooled prevalence was presented with 95% confidence intervals. Heterogeneity was assessed by calculating I² and Q value tests (I² > 50% or p < .01); the authors also critically evaluated the differences in the methodology of the articles.

Quality assessment

Three review authors (MF, RAS, and DS) independently and blindly assessed the methodological quality using the Joanna Briggs Institute (JBI) critical appraisal tools.²⁰ Two reviewers independently and blindly checked a list of questions for each study type and answered “yes,” “no,” or “unclear.” The discrepancies were resolved by review experts (EMD and IGD). The final score for each article was calculated by the number of yes in the total number of quality criteria. The score indicated the quality as low (<50%), average (50–75%), and high (>75%).

Results

The systematic search of the electronic databases resulted in 1873 articles; after removing duplicated articles, 1704 manuscripts remained for title and abstract screening. Afterward, 86 articles were considered eligible for full-text reading. The reviewers excluded 70 studies as they met the exclusion criteria of wrong population (n = 23), wrong intervention (n = 5), wrong outcome (n = 18), wrong study design (n = 19), foreign language (n = 3), and duplicate (n = 2) (Appendix C). In this systematic review, 16 studies were included for qualitative analysis and 15 for quantitative analysis (Figure 1).

Figure 1.

Flowchart summarizing the identification and selection process of the studies for inclusion. Adapted from Page et al., 2020.¹⁵

Characteristics of the studies

Most studies selected for this systematic review were conducted in the United States (n = 4).^21–24 The remaining studies were performed in various countries, including South Korea (n = 2),^25,26 Japan (n = 1),²⁷ Israel (n = 2),^28,29 Germany (n = 1),³⁰ Switzerland (n = 1),³¹ Belgium (n = 1),³² Spain (n = 1),³³ Italy (n = 1),³⁴ Sweden (n = 1),³⁵ and the Kingdom of Saudi Arabia (n = 1)³⁶ (Table 1). Although the search strategy included the time range between 2011 and 2023, the studies that met the inclusion criteria for this review were published between 2012 and 2022; the oldest ones were carried out by Polidori et al. (Italy)³⁴ and Fritz et al. (Switzerland)³¹ in 2012, and the most recent by Alsaidan et al. and Tukukino et al. in 2022.^35,36 The follow-up time of prescriptions made by physicians using a clinical physician order entry (CPOE) varied, with the shortest being just 7 days³⁰ and the longest being 46 months.²⁵ As for the study design, there was one longitudinal observation study,²⁶ ten cross-sectional studies,^{21,23–25,27,30,32,34–36} four prospective longitudinal studies,^22,29,31,33 and one study that performed retrospective cross-sectional and prospective longitudinal studies²⁸ (Table 1).

Table 1.

Summary of the characteristics of the included studies (n=16).

Author, Years, Country	Drug information database	Follow-up (days, month or year)	Setting	Outcomes evaluated	Prescription (N)	Alert (Number and percentage)	Override (number and percentage)	Conclusion
Ahn et al., (2014), Korea²⁵	Official governmental rules	46 months	Tertiary care hospital	Prevalence of pDDI alerts and alert overrides	14,780,519	51,864 (0.35%)	35,231 (67.9%)	pDDI alerts and the reasons for alert overrides differed by admitting department. Different strategies for each department and improving the alert rules may prevent inappropriate alerts.
Ahn et al., (2014), Korea²⁶	Official governmental rules	17 months	Tertiary care hospital	Prevalence of pDDI alerts and alert overrides	102,379	6,060 (5.9%)	4,409 (72.8%)	The risk of alerts and alert-override rates differed by admitting departments. The strategies to reduce DDI and alert overrides should consider the admitting department.
Alsaindan et al., (2022), Kingdom of Saudi Arabia ³⁵	Commercial (Cerner® Millennium Version 2014)	30 months	General care hospital	Prevalence of pDDI alert and alert override	5,114,310	318,809 (6.2%)	310,096 (97.3%)	The alert generation rate and override rate were considerably high for all alert types. Discovery of such high alert override rates, and poor documentation of reasons for overriding the alerts have been important for consideration to develop recommendations to improve medication related generation and alert handling with potential to improve clinical practice.
Amkreutz et al., (2017), Germany³⁰	Commercial software rules (MediQ and Meona)	7 days	Quaternary care Hospital	Prevalence of pDDI alerts generated by two different softwaresp	37,577	MediQ: 21,261 (56.5%)	NR	A high prevalence of pDDIs was detected by MediQ and Meona software. Medications should regularly be checked for pDDIs.
Amkreutz et al., (2017), Germany³⁰	Commercial software rules (MediQ and Meona)	7 days	Quaternary care Hospital		37,577	Meona: 7,681 (20.4%)	NR
Bhakta et al., (2019), USA²²	Commercial software rules (Epic Systems)	19 months (29 weeks pre-intervention and 52 weeks pos-tintervention)	Quaternary care Hospital	Incidence of pDDI alert and alert override before and after CDSS adjustment	Before: 68,900	Before: 30,600 (44%)	Before: 27,111 (88.5%)	A committee-led systematic approach to optimizing drug–drug interactions facilitated a significant decrease in the overall number of alerts and an increase in both medication alert acknowledge- ment and modification rates.
Bhakta et al., (2019), USA²²	Commercial software rules (Epic Systems)		Quaternary care Hospital		After: 50,300	After: 13,200 (26%)	After: 11,787 (89.3%)
Bryant et al., (2014), USA²¹	Commercial software rules (Cerner’s Multum knowledge base)	4 days	Two primary care hospitals	Prevalence of pDDI alerts and alert overrides	18,354	1,153 (6.28%)	1,097 (95%)	Despite intensive efforts to improve a commercial drug interaction alert system and to reduce alerting, override rates remain as high as reported over a decade ago. Alert fatigue does not seem to contribute. The results suggest the need to fundamentally question the premises of drug interaction alert systems.
Edrees, et al., (2020), USA²⁴	Commercial software rules (First Databank)	1 year	General care hospital	Prevalence of pDDI alert override	NR	16,011	15,318 (95.7%)	Our findings suggest an urgent need to implement steps to address and improve the effectiveness of these alerts. A first step is to evaluate and potentially turn off some of the less important alerts and offer stronger warnings for the most important ones.
Fritz, et al,. (2012), Switzerland³¹	Commercial software rules (Pharmavista, DrugReax and TheraOpt)	NR	Two tertiary care hospital	Prevalence of pDDI alert override using three different softwares	NR	Pharmavista: 53	Pharmavista: 50 (94.3%)	CDSS are valuable screening tools for medication errors, but only a small fraction of their alerts appear relevant in individual patients. In order to avoid overalerting CDSS should use patient-specific information and management-oriented classifications.
						DrugReax: 362	DrugReax: 345 (95.3%)
						TheraOpt: 328	TheraOpt: 315 (96%)
Knight, et al., (2019), USA²³	Commercial software rules (First Databank)	10 months (5 months before and after intervention)	General care Hospital	Prevalence os pDDI alert and alert override rate before and after CDSS adjustment	Before: 241,915	Before: 19,217 (8%)	Before: 18,819 (97.9%)	Decreasing alert burden and increasing the significance of DDI alerts presented to providers resulted in a statistically significant increase in the acceptance rate for medication alerts overall, and for DDIs in particular.
Knight, et al., (2019), USA²³	Commercial software rules (First Databank)	10 months (5 months before and after intervention)	General care Hospital		After: 245,757	After: 4,461 (2%)	After: 4286 (96%)
Muylle et al., (2021), Belgium³²	Commercial software rules (DelphiCare)	14 months (7 months before and after intervention)	General care Hospital	Prevalence of pDDI alert override before and after CDSS adjustment	NR	Before: 1,087	Before: 1,019 (94%)	The implementation of several CDSS optimization strategies was successful.
Muylle et al., (2021), Belgium³²	Commercial software rules (DelphiCare)	14 months (7 months before and after intervention)	General care Hospital		NR	After: 2,630	After: 1,959 (74%)
Nasuhara et al., 2015, Japan²⁷	Developed by the institution’s pharmacy department	1 year	General care Hospital	Prevalence of pDDI alerts and alert override	268,622	170 (0.06%)	111 (65.3%)	A computerized drug interaction alert system that requires password overrides appears useful for promoting medication safety. However, the password override system may represent an excessive burden for physicians and thereby increase alert fatigue.
Polidori, et al., (2012) Italy³⁴	Commercial software rules (Micromedex)	6 months	Quaternary care Hospital	Prevalence of pDDI alerts and monthly incidence of pDDI alert and adherence before and after CDSS adjustment	51,393	21,447 (41.7%)	Before (monthly)^a: ICU=30%, CSU= 59%, ASU= 57%, SDU= 64%, Dialysis= 71%, Endoscopy= 90%, PACU= 58%; after (monthly)^a: ICU= 5%, CSU= 5%, ASU= 5%, SDU= 6%, Dialysis= 4%, Endoscopy= 39%, PACU= 10%;	Improving DDI alerts will improve patient safety by more appropriately alerting clinicians.
						Before (monthly): ICU = 47%, CSU = 41%, ASU = 45%, SDU 39%, Dialysis = 43%, Endoscopy = 24%, PACU = 27%;
						After (Monthly): ICU= 50%, CSU= 42%, ASU= 42%, SDU 38%, Dialysis= 42%, Endoscopy= 22%, PACU= 32%;
Zenziper Straichman, et al., (2017), Israel²⁸	Commercial software rules (Lexi-Comp and Micromedex)	1 year	Tertiary care hospital	Prevalence of pDDI alerts and alert overrides from a retrospective (S1) and a prospective (S2) study.	S1: 390,841	S1: 145,103 (37%)	S1: 137,415 (94.7%)	The vast majority of SafeRx® alerts are overridden, and overriding is justified in most cases. Minimizing the number of alerts is essential to reduce the likelihood of developing “alert fatigue”.
Zenziper Straichman, et al., (2017), Israel²⁸	Commercial software rules (Lexi-Comp and Micromedex)	1 year	Tertiary care hospital		S2: 9,894	S2: 3,064 (30,9%)	S2: 2,934 (95.8%)
Tukukino et al., (2022), Sweden³⁵	Swedish national interaction database Janusmed	30 months	Two primary care Hospitals	Prevalence of pDDI alert override	NR	405	370 (81%)	Alerts can be expected to be presented in more than every other older patient with two or more drugs in their medication list. Most of the alerts were already being addressed in health care, for instance by the monitoring of clinical or laboratory parameters, or were not relevant in the clinical setting. In about one in ten alerts, however, it may be appropriate to take further action, although, for unknown reasons, such steps were not taken.
Velez-Diaz-Pallares, et al., (2013), Spain³³	Commercial software rules (Farmatools, Dominion Healthcare)	3 months	Tertiary care Hospital	Prevalence of pDDI related to vitamin K antagonists before and after CDSS implementation.	NR	Before: 46%	NR	Implementation of a CPOE interaction alert programme reduces interaction errors.
Velez-Diaz-Pallares, et al., (2013), Spain³³	Commercial software rules (Farmatools, Dominion Healthcare)	3 months	Tertiary care Hospital		NR	After: 22%	NR
Zenziper et al., (2014), Israel²⁹	Commercial software rules (Lexi-Comp and Micromedex)	6 months	Tertiary care hospital	Prevalence of CDSS alerts	136,459	21,799 (15.9%)^a	NR	To ensure its effectiveness, the implementation of a commercial CDSS in a large hospital requires system customization guided by the specific needs of particular departments, at both the initial and the maintenance stage. This task is best achieved by a multidisciplinary team in collaboration with the end-user.

^aEstimated by the authors; NR = Not reported; ICU = Intensive care unit; CSU = Cardiothoracic surgery unit; ASU = Abdominal surgery unit; SDU = step-down unit; PACU = Post-anesthesia care unit.

Regarding the studies’ research settings, two were conducted in three primary care hospitals,^21,35 six studies in a tertiary care hospital,^{25,26,28,29,31,33} three studies in a quaternary care hospital,^22,30,34 and five studies did not report the healthcare settings,^{23,24,27,32,36} so we counted them as general hospitals.

Among these healthcare settings, most employed commercial CDSS and drug information databases (n = 12),^{21–24,28–34,36} and four used a system and database, at least partly, developed by the public health system^25–27,35 (Table 1).

Main outcomes

Among the studies included in this systematic review, there were different types of reports on using CDSS to detect pDDIs in EHRs. Most of them evaluated the number of pDDI alerts generated by the CDSSs and the volume of alert overrides.^{21,23,25–28,36} Even in studies in which the period of the prevalence of alerts generated by the CDSS was below 10%, over 60% of these alerts were ignored by prescribers^{21,23,25–27,36} (Table 1).

Three other studies only evaluated the number of alerts for pDDIs and reported a high prevalence.^29,30,33 Amkreutz et al. (2017)³⁰ compared two software (MediQ and Meona) and found that both showed a high prevalence of DDI alerts (56,5% and 20,4%). Four other studies only evaluated the prevalence of alert overrides,^24,31,32,35 and one compared three software (Pharmavista, DrugReax, and TheraOpt).³¹ These studies also reported a high prevalence of alert overrides ranging from 74% to 96%. Notably, two studies evaluated the prevalence of alerts and alert overrides before and after adjusting the rules for alerts generated by the CDSS. However, despite reducing the number of generated alerts, these adjustments did not significantly reduce the number of overrides, in one of the studies it decreased from 97% to 96%,²³ and in the other it decreased from 94% to 74%.³² In addition, two studies reported the incidence of alerts generated and the volume of acceptance before and after adjustments in the drug interaction rules of these CDSS. In these studies, the number of alerts generated after CDSS adjustments was similar or lower than before, and the acceptance of these alerts increased after the intervention^22,34 (Table 1).

The authors of the studies differ about the reason for the high prevalence of alerts and alert override, most of them reporting that system adjustments should improve acceptance of alerts and decrease the chances of adverse drug events.^{22–26,28,29,31,32,34,36} Others emphasized that these adjustments must be constant, made by a multidisciplinary team assembled for this purpose, and take into account the characteristics of the hospital sector^25,26,29 and patient³¹ (Table 1).

Moreover, some researchers reported that only generating alerts for more important interactions can increase acceptance.^{22,24,23,28,35} Other studies only evaluated the generation of alerts and concluded that implementing a CDSS is essential to avoid DDIs.^30,33 One of the studies evaluated the use of a password for alert override and, due to the high rate of aversion, concluded that including an authentication step can increase the workload and generate alert fatigue.²⁷ In fact, one study concluded that the reason for the high levels of alert overrides is not related to professional fatigue but to the high number of alerts and that the real reason should be further investigated²¹ (Table 1).

Meta-analysis

Of the 16 studies in the systematic review, 15 were included for quantitative analysis. Meta-analyses were performed separately for the subgroups, which were divided into studies that only dealt with generated pDDI alerts and used the number of alert substitutions. Ten studies were included in the meta-analysis of the prevalence of alerts generated by the CDSSs, considering that 21,435,597 prescriptions were analyzed. The overall prevalence obtained was 13.7% (CI95% 5.6–24.7%, p-value <.0001, I^2 = 100%).^{21,23,25–30,34,36} Among them, one study compared two different software,³⁰ one study presented the prevalence of alerts generated before and after CDSS adjustments,²³ and another study consisted of two parts, one retrospective and one prospective.²⁸ Therefore, the respective prevalences appear in the meta-analysis (Figure 2).

Figure 2.

Prevalence of alerts generated by the CDSS: 21,435,597 prescriptions were analyzed, and the overall prevalence obtained was 13.7% (CI95% 5.6–24.7%, p-value <0.0001, I^2 = 100%). Among them, a study compared two different software,³⁰ a study presented the prevalence of alerts generated before and after CDSS adjustments,²³ and another study consisted of two parts, one retrospective and one prospective.²⁸ Therefore, the respective prevalences appear in the meta-analysis.

Regarding physicians’ adherence to receiving a pDDI alert in their prescription, eleven studies assessed the prevalence of alert overrides; 570,776 prescriptions were analyzed, and the overall prevalence obtained was 90% (CI95% 85.6–95.0%, p-value <.0001, I^2 = 100%) (Figure 3).^{21,23–28,31,32,35,36} Among them, two studies assessed the prevalence of disregard for alerts before and after adjustments to alert definitions in the CDSS,^23,32 one study compared three different software,³¹ and another was divided into two stages, one retrospective and one prospective.²⁸ Therefore, the respective prevalences appear in the meta-analysis (Figure 3).

Figure 3.

Prevalence of alert overrides by physicians: 570,776 prescriptions were analyzed, and the overall prevalence obtained was 90% (CI95% 85.6–95.0%, p-value studies assessed the prevalence of disregard for alerts before and after adjustments to alert definitions in the CDSS,^23,32 one study compared three different software,³¹ and another study was divided into two stages, one retrospective and one prospective.²⁸ Therefore, the respective prevalences appear in the meta-analysis.

Quality assessment

Most studies demonstrated high methodological quality,^{21,23–27,29,32,33,35,36} one moderate,²² and one low methodological quality³¹ (Supplementary File S2). The questions that most indicated low methodological quality were questions 3 and 8: “(3) Was the sample size adequate?”,^22,31,33,35 and “(8) Was there appropriate statistical analysis?”.^{27,29,30,33,35} The main issues related to low methodological quality assessment across studies were sample size and statistical analysis.

Discussion

Studies have shown that most of the alerts generated at the time of prescription, indicating pDDIs, are ignored by physicians. Although there is no consensus on the ideal number of alerts that should be generated, it is already known that many inappropriate alerts can reduce users’ confidence in the system.³⁷ In this context, Wickens and Dixon (2007)³⁸ reviewed the literature in search of indirect evidence on this topic to establish a diagnostic reliability value below which automation becomes useless or even worse than the performance before its implementation. This analysis revealed that reliability of 70% was the “cutoff point” below which automation was worse than no automation.³⁸

In this review, we evaluated the frequency of alerts generated by these CDSS when physicians prescribe medication for their patients and found a prevalence of 13.7%. We could consider a low value, given the number of drugs that interact with each other and the difficulty of the prescriber remembering all of them. However, even though this tool was designed to help the prescriber, we observed that these physicians ignored the pDDI alerts generated by these CDSS, where the prevalence of alert override was 90%. Thus, it is evident that these systems require adjustments so that the adherence to their pDDI indications increases and the number of alert overrides decreases (Figure 4).

Figure 4.

The methodological quality of the studies was qualified according to the percentage of “Yes” responses, where: Low < 50%; moderate= 50<75%; High = 75-100%. Y= yes, N = no, U = unclear and NA = not applicable.

This is an important finding, which shows that despite being informed about pDDI, prescribers choose to ignore this information and subject the patient to possible ADE. However, some studies included in this review evaluated the consequences of alert override, where after the dispensing and administration of prescribed medications with pDDI, the authors checked whether ADE had occurred. Interestingly, these studies reported a low prevalence, ranging from 4%³¹ to 7,8%,²⁴ or no occurrence of ADE²⁷ in the location where these studies were performed. However, it is important to remember that as these are studies carried out in hospitals, patients are monitored full-time, so physicians may have chosen to generate a greater benefit than the risk offered, as these patients would have all the necessary support in case of ADE. Thus, we could assume that pDDI alerts generated by CDSS would not represent a real risk to patients’ health, but when we extrapolate this data and apply this prevalence to a high number of patients, who often do not have the support of a hospital to manage this risk, we can consider CDSS as important tools to avoid ADE with serious consequences for patients.

Some studies have reported reasons why physicians override alerts. Among these most common reasons we can mention that prescribing doctors often consider that the benefit obtained from the administration of prescribed medications will be greater than the risk of developing an ADE.^25,28,31 Furthermore, prescribers often considered the alert to have low clinical relevance and therefore overrode the alerts.^25,28 In the latter case, it is important to highlight that many CDSS did not consider the patients’ laboratory tests, nor the route of administration, which directly influences the clinical relevance of the pDDI alert. But the most cited reason in one of the studies was “Other”, where prescribing doctors did not even indicate the reason for cancellation, which may indicate negligence on the part of the doctor.²⁷

Another issue to consider is the quantity and quality of these alerts; alerts of lesser clinical importance, when generated in excess, may tire these prescribers, also known as alert fatigue. This phenomenon has already been reported previously, showing that this problem can cause physicians to start ignoring these alerts after an excess of generated alerts, increasing the risk of alerts with greater clinical importance going unnoticed.³⁹ Some studies included in this review evaluated the effect of adjustments in the rules of the clinical decision support system to reduce the number of alerts generated. After these adjustments, only alerts of greater clinical importance were generated; with this, the number of alerts decreased significantly, although the acceptance of these alerts did not increase significantly, and physicians continued to override most of these alerts.^{21–23,32,34} These findings contribute to demonstrating the complexity of improving the acceptance of pDDI alerts. Among the studies that demonstrated the lowest alert override rates, we can highlight the studies by Ahn et al., (2014),^25,26 which were conducted in South Korea, where the Korean government does not usually reimburse the value of prescribed medications that had pDDI alerts, so prescribers tend to pay more attention to the alerts. Another study that found a lower than average prevalence of alert override is the study conducted by Nasuhara et al., (2015),²⁷ where the authors report that because physicians had to consult directly with pharmacists to obtain override passwords, physicians could only decide to continue with their orders after receiving detailed pharmaceutical reasons for the contraindications.²⁷

After providing evidence-based information and removing minor alerts, a CDSS requires rigorous evaluation to determine the optimal sensitivity and specificity ratio to reduce patient harm. No system can achieve 100% sensitivity and specificity in a real-world setting.⁴⁰ Filtering lists of drug interactions in CDSS databases, keeping only clinically significant pairs, may improve the “alert fatigue” effect but also create liability concerns for clinicians, who could perceive these systems to be at risk of making mistakes. However, using a list of DDI based on consensus between professional societies or relevant regulatory bodies could increase confidence in these systems.⁴¹

To reduce the number of alerts and increase their clinical relevance, the CDSS should not be used as an independent system but to work with the EHR and cross-check important patient information, such as laboratory test results, information on comorbidities, and other clinical parameters. Therefore, about 30% of alerts could be avoided if only five laboratory test results were integrated into the system, including potassium, white blood cell count, international normalized ratio, therapeutic drug monitoring, and glomerular filtration rate values.³⁶ Another way to reduce the number of alerts is to direct these alerts based on the specialty of the prescribing physician, for instance, not generating an excessive number of renal risk alerts for a kidney specialist with many years of experience.⁴⁰ With this, confidence in these systems tends to increase since physicians ignore alerts because of their lack of specificity; thus, alerts generated for the general population could be changed if the characteristics of patients and physicians were considered.³⁶

In this context, two literature reviews sought to analyze the reasons why doctors accept or do not accept the implementation of a CDSS.^41,42 These reviews were conducted in different decades, however, their findings show similar results, where the authors report the barriers and facilitators in the acceptance of this technology, which represents a change in physicians’ routine. These reviews did not specifically focus on CDSS that generated drug interaction alerts and included other types of alerts related to medication prescriptions. However, we can use the findings of these studies as a basis to better understand the reason for the high prevalence of alert override shown in this study. As barriers to the acceptance of CDSS, the authors cite poor interface design, inadequate implementation (e.g. without prior training) and inadequate data maintenance. Furthermore, the authors of these studies report that for physicians to accept CDSS, they need to be easy to use, efficient, present clear and objective alerts and contain relevant information. It is worth noting that the definition of relevant information is subjective, as the same information may be relevant to one physician and not to another. Therefore, the authors suggested that physicians be involved during the development of these systems and that user-centered design may be a suitable method.⁴²

The strength of the present systematic review is consolidated in the following points: (a) analysis by paired blinded reviewers, (b) exhaustion of the literature search, (c) data validation, (d) search for expertise, and (e) quality assessment. Nonetheless, this study also had some limitations, including unclear outcomes in some of the analyzed articles; hence, they were estimated by the present article’s reviews. In addition, the variation in alert prevalence and alert override prevalence can be attributed to several factors. First, the rules of alerts issued by the systems varied considerably, and most studies did not provide information about this. Secondly, the data collection period was limited in most studies, where more than half of the included studies were conducted for less than a year. Third, institutions with different levels of health care complexity may value certain DDI as more important or relevant to the populations they serve, thus prioritizing a specific group of DDI alerts over others. Another limitation of this review was that the search strategy was limited to a time range between 2011 and 2023. This may have resulted in the low number of studies included, given that the subject is important and that many other studies on the topic may have been executed.

Conclusion

Our results show that prescribers ignore most of the alerts these clinical decision support systems generate since 90% of physicians override them. Even after adjustments were made to these systems to reduce the number of alerts and avoid professional fatigue, the number of alert overrides did not decrease satisfactorily. Therefore, these systems should be developed taking into account the particularities of each institution where this system will be implemented, as well as the experience and clinical specialty of the physician who will use this system. This could increase the users’ confidence and satisfaction with the CDSS and possibly decrease alert overrides, improving the clinical safety of the treatments offered to patients. Furthermore, integrated systems, where prescriptions take into account laboratory findings as well as individual characteristics of each patient, could also improve the acceptance of alerts by prescribers. In this systematic review with meta-analysis, we showed that despite the CDSS being recognized as an important tool to prevent adverse events related to drugs prescribed by physicians in health units, this instrument has been underused, wasting users’ time and money. Therefore, in some cases, to reduce the alert override rates, it may be necessary to apply more drastic solutions, such as not reimbursing the institution where the drugs with pDDI were prescribed.

Supplemental Material

Supplemental Material - Override rate of drug-drug interaction alerts in clinical decision support systems: A brief systematic review and meta-analysis

Supplemental Material for Override rate of drug-drug interaction alerts in clinical decision support systems: A brief systematic review and meta-analysis by Mariano Felisberto, Geovana dos Santos Lima, Ianka Cristina Celuppi, Miliane dos Santos Fantonelli, Wagner Luiz Zanotto, Júlia Meller Dias de Oliveira, Eduarda Talita Bramorski Mohr, Ranieri Alves dos Santos, Daniel Henrique Scandolara, Célio Luiz Cunha, Jades Fernando Hammes, Júlia Salvan da Rosa, Izabel Galhardo Demarchi, Raul Sidnei Wazlawick, and Eduardo Monguilhott Dalmarco in Health Informatics Journal.

Supplemental Material

Supplemental Material - Override rate of drug-drug interaction alerts in clinical decision support systems: A brief systematic review and meta-analysis

Supplemental Material

Supplemental Material - Override rate of drug-drug interaction alerts in clinical decision support systems: A brief systematic review and meta-analysis

Footnotes

Authors’ contributions

MF, GSL, ICC, JMDO, ETBM, JSR, CLC, and JFH performed the bibliographic search, and IGD and EMD validated it. MF, RAS, DHS, and ICC participated in elaborating the data extraction table. EMD, IGD, and RSW analyzed and critically reviewed the data. MF and JMDO performed the meta-analysis. MF, EMD, and IGD wrote the general text, and each stage was submitted for analysis and review by the co-authors, with their due criticism. MF, RAS, DHS, JMDO, ETBM, and IGD evaluated the quality assessment. IGD and EMD critically reviewed the drafts and subsequent steps. All authors approved the final version of the manuscript for submission. All authors had full access to all data in the study and assumed responsibility for the data’s integrity and data analysis accuracy.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior [Coordination for the Improvement of Higher Education Personnel] – Brazil (CAPES) – Finance Code 001, and by Brazilian Ministry of Health (e-SUS PHC Project Stage 4). The RSW and EMD are productivity fellow in technology development and innovative extension of CNPq.

Ethical statement

Registration and protocol

This systematic review protocol was based on PRISMA-P and registered in the PROSPERO International Prospective Register of Systematic Reviews under registration number CRD42021261967.

ORCID iD

Eduardo Monguilhott Dalmarco

Supplemental Material

Supplemental material for this article is available online.

References

Assiri

Shebl

Mahmoud

, et al. What is the epidemiology of medication errors, error-related adverse events and risk factors for errors in adults managed in community care contexts? A systematic review of the international literature. BMJ Open 2018; 8(5): e019101. DOI: 10.1136/bmjopen-2017-019101.

Kohn

Corrigan

Donaldson

(eds). Institute of Medicine (US) Committee on Quality of Health Care in America, To Err is Human: Building a Safer Health System. National Academies Press (US), 2000. DOI: 10.17226/9728.

Krähenbühl-Melcher

Schlienger

Lampert

, et al. Drug-related problems in hospitals: a review of the recent literature. Drug Saf 2007; 30(5): 379–407. DOI: 10.2165/00002018-200730050-00003.

Wishart

Feunang

Guo

, et al. DrugBank 5.0: a major update to the DrugBank database for 2018. Nucleic Acids Res 2018; 46(D1): D1074–D1082. DOI: 10.1093/nar/gkx1037.

Van De Sijpe

Quintens

Walgraeve

, et al. Overall performance of a drug-drug interaction clinical decision support system: quantitative evaluation and end-user survey. BMC Med Inf Decis Making 2022; 22(1): 48. DOI: 10.1186/s12911-022-01783-z.

Zheng

Richardson

, et al. Drug-drug interactions and their harmful effects in hospitalised patients: a systematic review and meta-analysis. Eur J Clin Pharmacol 2018; 74(1): 15–27. DOI: 10.1007/s00228-017-2357-5.

Feng

Wei

. An empirical study on user experience evaluation and identification of critical UX issues. Sustainability 2019; 11(8): 2432, DOI: 10.3390/su11082432.

Baysari

Van Dort

Stanceski

, et al. Is evidence of effectiveness a driver for clinical decision support selection? A qualitative descriptive study of senior hospital staff. Int J Qual Health Care 2023; 35(1): mzad004. DOI: 10.1093/intqhc/mzad004.

Luna

Rizzato Lede

Otero

, et al. User-centered design improves the usability of drug-drug interaction alerts: experimental comparison of interfaces. J Biomed Inf 2017; 66: 204–213. DOI: 10.1016/j.jbi.2017.01.009.

10.

Humphrey

Mirica

Phansalkar

, et al. Clinician perceptions of timing and presentation of drug-drug interaction alerts. Appl Clin Inf 2020; 11(3): 487–496. DOI: 10.1055/s-0040-1714276.

11.

Kaushal

Shojania

Bates

. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003; 163(12): 1409–1416. DOI: 10.1001/archinte.163.12.1409.

12.

Nuckols

Smith-Spangler

Morton

, et al. The effectiveness of computerized order entry at reducing preventable adverse drug events and medication errors in hospital settings: a systematic review and meta-analysis. Syst Rev. 2014;3:56. DOI:10.1186/2046-4053-3-56

13.

Prgomet

Niazkhani

, et al. Impact of commercial computerized provider order entry (CPOE) and clinical decision support systems (CDSSs) on medication errors, length of stay, and mortality in intensive care units: a systematic review and meta-analysis. J Am Med Inf Assoc 2017; 24(2): 413–422. DOI: 10.1093/jamia/ocw145.

14.

Korb-Savoldelli

Boussadi

Durieux

, et al. Prevalence of computerized physician order entry systems-related medication prescription errors: a systematic review. Int J Med Inform 2018; 111: 112–122. DOI: 10.1016/j.ijmedinf.2017.12.022.

15.

Page

McKenzie

Bossuyt

, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021; 372: n71. DOI: 10.1136/bmj.n71, Published 2021 Mar 29.

16.

Clarivate Analytics . EndNote X9 computer program, 2021, https://support.clarivate.com/Endnote/s/article/EndNote-X9?language=enUS (accessed 10 July 2021).

17.

Ouzzani

Hammady

Fedorowicz

, et al. Rayyan-a web and mobile app for systematic reviews. Syst Rev 2016; 5(1): 210. DOI: 10.1186/s13643-016-0384-4.

18.

Urbaniak

Plous

. (2013). Research randomizer (Version 4.0) [Computer software]. Retrieved from: http://www.randomizer.org/ (accessed 15 july 2021).

19.

R Core Team . R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing, 2020. RStudio 2022.12.0+353 https://www.R-project.org/

20.

Joanna Briggs Institute

JBI

. Critical appraisal tools 2020. Adelaide, Australia: The University of Adelaide. Available at: https://www.jbi.global/critical-appraisal-tools# (Accessed 10 March 2021).

21.

Bryant

Fletcher

Payne

. Drug interaction alert override rates in the Meaningful Use era: no evidence of progress. Appl Clin Inf 2014; 5(3): 802–813. DOI: 10.4338/ACI-2013-12-RA-0103, Published 2014 Sep 3.

22.

Bhakta

Colavecchia

Haines

, et al. A systematic approach to optimize electronic health record medication alerts in a health system. Am J Health Syst Pharm 2019; 76(8): 530–536. DOI: 10.1093/ajhp/zxz012.

23.

Knight

Maygers

Foltz

, et al. The effect of eliminating intermediate severity drug-drug interaction alerts on overall medication alert burden and acceptance rate. Appl Clin Inf 2019; 10(5): 927–934. DOI: 10.1055/s-0039-3400447.

24.

Edrees

Amato

Wong

, et al. High-priority drug-drug interaction clinical decision support overrides in a newly implemented commercial computerized provider order-entry system: override appropriateness and adverse drug events. J Am Med Inf Assoc 2020; 27(6): 893–900. DOI: 10.1093/jamia/ocaa034.

25.

Ahn

Cho

Shin

, et al. Differences of reasons for alert overrides on contraindicated Co-prescriptions by admitting department. Healthc Inform Res 2014; 20(4): 280–287. DOI: 10.4258/hir.2014.20.4.280.

26.

Ahn

Kam

Park

, et al. Differences among admitting departments in alerts and alert overrides for drug-drug interaction. Pharmacoepidemiol Drug Saf 2014; 23(4): 390–397. DOI: 10.1002/pds.3591.

27.

Nasuhara

Sakushima

Endoh

, et al.

Physicians’ responses to computerized drug interaction alerts with password overrides

[published correction appears in BMC Med Inform Decis Mak. 16(1):74]. BMC Med Inf Decis Making. 2015;15:74. Published 2015 Aug 28. DOI:10.1186/s12911-015-0194-y

28.

Zenziper Straichman

Kurnik

Matok

, et al. Prescriber response to computerized drug alerts for electronic prescriptions among hospitalized patients. Int J Med Inform 2017; 107: 70–75. DOI: 10.1016/j.ijmedinf.2017.08.008.

29.

Zenziper

Kurnik

Markovits

, et al. Implementation of a clinical decision support system for computerized drug prescription entries in a large tertiary care hospital. Isr Med Assoc J 2014; 16(5): 289–294.

30.

Amkreutz

Koch

Buendgens

, et al. Prevalence and nature of potential drug-drug interactions among kidney transplant patients in a German intensive care unit. Int J Clin Pharm 2017; 39(5): 1128–1139. DOI: 10.1007/s11096-017-0525-4.

31.

Fritz

Ceschi

Curkovic

, et al. Comparative evaluation of three clinical decision support systems: prospective screening for medication errors in 100 medical inpatients. Eur J Clin Pharmacol 2012; 68(8): 1209–1219. DOI: 10.1007/s00228-012-1241-6.

32.

Muylle

Gentens

Dupont

, et al. Evaluation of an optimized context-aware clinical decision support system for drug-drug interaction screening. Int J Med Inform 2021; 148: 104393. DOI: 10.1016/j.ijmedinf.2021.104393.

33.

Vélez-Díaz-Pallarés

Ruano

Villamañán

, et al. Impact of electronic prescription alerts on medication errors related to vitamin K antagonists in hospitalised patients. Eur J Hosp Pharm 2014; 21: 29–33. DOI: 10.1136/ejhpharm-2013-000308.

34.

Polidori

Di Giorgio

Provenzani

. Incidence of potential drug interactions in a transplant centre setting and relevance of electronic alerts for clinical practice support. Inf Prim Care 2012; 20(4): 257–262. DOI: 10.14236/jhi.v20i4.15.

35.

Tukukino

Parodi López

Svensson

, et al. Drug interaction alerts in older primary care patients, and related medically justified actions. Eur J Clin Pharmacol 2022; 78(7): 1115–1126. DOI: 10.1007/s00228-022-03292-4.

36.

Alsaidan

Portlock

Ata

, et al. Retrospective descriptive assessment of clinical decision support medication-related alerts in two Saudi Arabian hospitals. BMC Med Inf Decis Making 2022; 22(1): 101. DOI: 10.1186/s12911-022-01838-1, Published 2022 Apr 15.

37.

Seidling

Klein

Schaier

, et al. What, if all alerts were specific - estimating the potential impact on drug interaction alert burden. Int J Med Inform 2014; 83(4): 285–291. DOI: 10.1016/j.ijmedinf.2013.12.006.

38.

Wickens

Dixon

. The benefits of imperfect diagnostic automation: a synthesis of the literature. Theor Issues Ergon Sci 2007; 8: 201–212. DOI: 10.1080/14639220500370105.

39.

Park

Chae

Jeong

, et al. Appropriateness of alerts and physicians’ responses with a medication-related clinical decision support system: retrospective observational study. JMIR Med Inform 2022; 10(10): e40511. DOI: 10.2196/40511.

40.

Poly

Islam

Yang

, et al. Appropriateness of overridden alerts in computerized physician order entry: systematic review. JMIR Med Inform 2020; 8(7): e15653. DOI: 10.2196/15653.

41.

Horsky

Phansalkar

Desai

, et al. Design of decision support interventions for medication prescribing. Int J Med Inform 2013; 82(6): 492–503. DOI: 10.1016/j.ijmedinf.2013.02.003.

42.

Westerbeek

Ploegmakers

de Bruijn

, et al. Barriers and facilitators influencing medication-related CDSS acceptance according to clinicians: a systematic review. Int J Med Inform 2021; 152: 104506. DOI: 10.1016/j.ijmedinf.2021.104506.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.39 MB

0.46 MB

0.21 MB