Sage Journals: Discover world-class research

Abstract

A CDSS generates a high number of inappropriate alerts that interrupt the clinical workflow. As a result, clinicians silence, disable, or ignore alerts, thereby undermining patient safety. Therefore, the effectiveness and appropriateness of CDSS alerts need to be evaluated. A systematic review was carried out to identify the factors that affect CDSS alert appropriateness in supporting clinical workflow. Seven electronic databases (PubMed, Scopus, ACM, Science Direct, IEEE, Ovid Medline, and Ebscohost) were searched for English language articles published between 1997 and 2018. Seventy six papers met the inclusion criteria, of which 26, 24, 15, and 11 papers are retrospective cohort, qualitative, quantitative, and mixed-method studies, respectively. The review highlights various factors influencing the appropriateness and efficiencies of CDSS alerts. These factors are categorized into technology, human, organization, and process aspects using a combination of approaches, including socio-technical framework, five rights of CDSS, and Lean. Most CDSS alerts were not properly designed based on human factor methods and principles, explaining high alert overrides in clinical practices. The identified factors and recommendations from the review may offer valuable insights into how CDSS alerts can be designed appropriately to support clinical workflow.

Keywords

alert fatigue appropriateness clinical decision support systems clinical workflow Lean

Introduction

The Institute of Medicine advocates that health information technologies (HITs), including electronic clinical decision support systems (CDSSs), can improve and enhance the quality of care.¹ A CDSS, with its alert function, can be integrated into other health information systems (HISs), such as electronic medical records (EMRs) and computerized provider order entry (CPOE), to enhance the decision making of clinicians by providing relevant information at the point of care.² A CDSS uses different notification methods, such as alerts, pop-ups, reminders, and screen flashes, to inform clinicians about dangerous drug–drug interactions (DDIs), drug allergies, drug duplication, and drug doses that may lead to adverse events and medication errors.^2–4 Medication alerts remain an important component of CDSSs because they provide evidence-based recommendations to users.⁵ However, clinicians encounter too many medication alerts when prescribing, reviewing, dispensing, and administering medications, which disrupt their workflow.^6,7

Factors contributing to inappropriate alerts include lack of data granularity in CDSS knowledge base, poorly categorized DDI alerts, low alert specificity/sensitivity, and perceived risk of legal liability.^4,5,8,9 Most alerts triggered during medication order entry are overridden or disregarded because they display inappropriate information and unclear warnings.^9,10 Clinicians should respond to alerts appropriately to realize the full potential of CDSS in improving the quality of care and patient safety.^4,11 Appropriate CDSS alerts are alerts that are presented to users in the right context and at the right time, less intrusively displayed on the screen, and display clear, correct, and useful patient information.^11,12 Therefore, the appropriateness of CDSS alerts should be evaluated to ensure proper design, implementation, and configuration to support clinical processes.

Theoretical background

A CDSS generates many inappropriate alerts, which explains why most clinicians override alerts.¹⁰ Increasing rates of alert override denote that alerts impair the judgments of clinicians and do not assist clinicians in making timely, informed, and high-quality decisions.^9,13 The five rights of CDSS state that CDSS interventions such as alerts should be designed to provide the right information to the right person in the right format through the right channel at the right time during task execution.¹⁴ Achieving the five rights for CDSSs is challenging because of several socio-technical aspects, including poor CDSS design, unreliable hardware and software, interface usability, and data integrity, accessibility, and security.^15–17

According to Wasylewicz et al.,¹⁸ understanding how to alert clinicians at the point of care is a complex socio-technical issue that requires consideration of technological, human, process, and organizational aspects. We proposed socio-technical dimensions and measures for CDSS alert appropriateness based on a critical review of selected models, namely, human–organization—technology–fit,¹⁹ human-automation and interaction,²⁰ business process management (BPM) life cycle,²¹ systems engineering initiative for patient safety,²² and previous works on CDSS alert and workflow problems (Table 1). The models were selected based on their inclusion of socio-technical interactions among human agents, tools, technology, and their environment.^23,24 We argue that considering technology, human, organization, and process and the fit among them can increase alert effectiveness and appropriateness, reducing alert overrides (Figure 1) and subsequent compliance to the five CDSS rights.

Table 1.

Example of evaluation measures for CDSS alert appropriateness.

Technology	Human	Organization	Process	Outcome
System design Alert sensitivity/specificityAlert timingAlert presentationInterface usabilityFalse alertAlert contentAlert overloadPoor signal/noise ratioKnowledge baseSystem implementation Hardware and softwareNetworkWorkload	TrainingKnowledgeSkillsEducationAttitudePerceptionBeliefBehaviorResponse	Policy and procedure PrivacySecurityConfidentialityStandardsRules and regulations	Clinical task Prescribing, dispensingProcedure ordersRefillingBillingReviewing alertsDocumentationCommunicationProcess analysis Waste (e.g. delay and error)Inefficient processesProcess redesign Workflow efficiencyMappingProcess implementation ModificationAutomationProcess monitoring/ improvement BottlenecksRecurrent errorsPhysician productivityTuning and optimization	Non-interruptive alertsLess intrusive alertsAccurate informationReduced error and adverse drug eventsPatient satisfaction

Figure 1.

Considering and aligning socio-technical aspect for CDSS alert.

Workflow disruption largely causes clinicians to ignore, misinterpret, and mishandle drug alerts.^16,25,26 Alerts that support clinical workflow are likely to be used.^4,5,8,9,26 The assessment of workflow and its fit with CDSS alert should be carried throughout CDSS development and post implementation.²⁶ Workflow assessment can be carried out using established process-oriented methods, such as BPM and Lean, to identify barriers to the fit of CDSS alert and workflow. BPM is the art and science of overseeing how work is performed in an organization to ensure consistent outcomes and to take advantage of improvement opportunities.²¹ Typical examples of improvement include reducing cost, execution time, error rates, and waiting times. BPM encompasses various tools and methods that are used to analyze, model, simulate, monitor, and execute end-to-end business processes.²⁷ BPM can be combined with other process improvement methods, such as Lean, to improve or enhance business processes.²¹

Lean methods are designed to improve processes by eliminating non-value added (NVA) activities known as waste.^12,28,29 Waste in HIS can be defined as NVA activities in the work process that hinders useful information to its users.³⁰ Lean categorizes eight types of waste as defects, overproduction, transportation, waiting, inventory, motion, over processing, and underutilized human potential.^12,28,29 The waste model has been applied and extended to identify different types of waste in healthcare, including overtreatment; failures in the coordination and execution of care processes; administrative complexity; and pricing failure, fraud, and abuse.^31,32 This review used the waste model to analyze workflow inefficiencies related to alerts. The wide application of Lean and BPM in healthcare is attributed to its ability to analyze and redesign clinical process and address the challenges of complying to the five rights of CDSS. This review aims to identify factors affecting the appropriateness of CDSS alerts in supporting clinical workflow based on the proposed evaluation measures for CDSS alerts to illustrate the factors in compliance to the five rights, inefficiencies, and complexity, which are also featured in the measures.

Methodology

We followed the guidelines of Preferred Reporting Items for Systematic Reviews and meta-analyses in reporting our systematic review (Figure 2). We registered the review on the Prospero database; however, the review slightly deviated from the Prospero protocol by applying the five-right, socio-technical, and Lean perspective. The search was conducted on 11 October 2018 to 28 December 2018 by using the following keywords: computerized provider order entry, CPOE, alert, reminder, pharmacy information systems, PIS, clinical decision support system, CDSS, alert fatigue, alert override, electronic health record, EHR, electronic medical record, EMR, health information system, HIS, prompt, notification, health information technology, HIT, clinical workflow, clinical process, patient safety, medical errors, adverse events, business process management, BPM, process mapping, and process re-engineering (Table 2). The electronic search was performed on seven bibliographic databases, namely, PubMed, Scopus, ACM, Science Direct, IEEE, Ovid Medline, and Ebscohost, for relevant literature from 1997 to 2018.

Figure 2.

Study selection.

Table 2.

Search terms used for bibliographic database.

Search strings
(“computerize provider order entry” OR “CPOE” OR “alert” OR “reminder” OR “electronic health record” OR “EHR” OR “pharmacy information systems” OR “PIS”) AND (“clinical decision support system” OR “CDSS” OR “alert fatigue” or “alert override” OR “electronic health record” OR “EHR” OR “electronic medical record” OR “EMR”) AND (“health information system” OR “HIS” OR “prompt” OR “notification” OR “health information technology” OR “HIT” OR “clinical workflow” OR “clinical process” OR “patient safety” OR “medical errors” OR “adverse events” OR “Process management” OR “process mapping” OR “process re-engineering”)

Search strings

(“computerize provider order entry” OR “CPOE” OR “alert” OR “reminder” OR “electronic health record” OR “EHR” OR “pharmacy information systems” OR “PIS”) AND (“clinical decision support system” OR “CDSS” OR “alert fatigue” or “alert override” OR “electronic health record” OR “EHR” OR “electronic medical record” OR “EMR”) AND (“health information system” OR “HIS” OR “prompt” OR “notification” OR “health information technology” OR “HIT” OR “clinical workflow” OR “clinical process” OR “patient safety” OR “medical errors” OR “adverse events” OR “Process management” OR “process mapping” OR “process re-engineering”)

Article screening and quality assessment were carried out by one reviewer (OO). We seek to include comprehensive studies, regardless of their methods, that are related to CDSS appropriateness to identify and understand their influencing factors. Articles were included if the title and abstracts indicated that they are primary research studies that employ qualitative, quantitative, retrospective cohort, or mixed methods focusing on basic and advance medication alerts³³ embedded in CDSSs. Furthermore, full texts of relevant articles reporting on features and functions pertinent to CDSS alert appropriateness were identified by including articles reporting on one or more rights in the five rights principles,³⁴ evaluation measures for CDSS alert appropriateness (Table 1), and medication alert- and workflow-related problems in healthcare settings. Studies on medical device alarms,³⁵ such as physiological monitors, ultrasonic humidifiers, infusion pumps, ventilators, and defibrillators, were excluded. Reports, reviews, editorials, commentaries, and non-English articles were also excluded. Citation chaining of included articles was performed to obtain additional relevant publications.

The bibliographic databases returned 9778 publications. The numbers of articles retrieved from each database are as follows: Scopus (n = 1009), Science Direct (n = 1511), IEEE digital library (n = 655), ACM (n = 2082), Ovid Medline (n = 249), Ebscohost (n = 2511), and PubMed (n = 1752). Duplicates were removed (n = 872). Then, 8906 titles and abstracts were initially screened based on their title and abstracts, and 8802 were excluded due to the exclusion reasons stated in Figure 2. Upon full-text screening records, only 104 articles were accessed, and 28 papers were further excluded because of their irrelevance to the research question, as listed in Figure 2.

One reviewer (OO) used data extraction forms to extract specific information regarding the study location, objectives, types of CDSS, country, methods, and key findings. Each paper went through the iterative process of interpreting and coding. Deductive reasoning was conducted based on the identified themes of the proposed evaluation measures for CDSS alert appropriateness, whereas inductive reasoning was applied to new, emerging themes. Findings were synthesized narratively and tabulated based on the overall themes. We employed two quality assessment tools, namely, Critical Appraisal Skill Program (CASP)³⁶ and Mixed Methods Appraisal Tool (MMAT).³⁷ Both tools can be used for systematic reviews that include qualitative, quantitative, and mixed methods studies,^36,37 but they slightly differ in terms of quantitative design types. Systematic reviews of mixed studies, such as ours, can benefit from these hybrid appraisal tools. We used a yes or no checklist to assess the quality of selected papers on the basis of the questions in Table 3 for qualitative and retrospective studies.³⁷ The assessment shows that majority of the selected papers are of high quality. In addition, MMAT is used for assessing the quality of quantitative (survey) and mixed method studies (Table 3).³⁷ The quality assessment reveals that over 73 of the papers are of high study quality, and the remaining three papers have acceptable study quality.

Table 3.

Critical appraisal for selected studies.

Tool/study type	Question
CASP for qualitative studies	Is the statement of the research aim clear?	Is the qualitative methodology appropriate?	Is the research design appropriate to address the aim of the research?
CASP for retrospective studies	Does the study address a clearly focused issue?	Is the cohort recruited in an acceptable way?	Is the outcome accurately measured to minimize bias?
MMAT for quantitative (survey) studies	Is the sampling strategy relevant to address the research question?	Are the measurements appropriate?	Is the statistical analysis appropriate to answer the research question?
	Is the sample representative of the target population?
	Is the risk of nonresponse bias low?
MMAT for mixed method studies	Is the research design appropriate?	Do the qualitative and quantitative data have proper integration?	Is the consideration on limitations associated with this integration appropriate?

Results

In summary, 76 papers were obtained (64 main papers and 14 additional papers from citation chaining) (Appendix A), of which 26 papers are retrospective cohort studies, 24 are qualitative, 15 are quantitative, and 11 are mixed-method studies. These studies were conducted in different settings of inpatient and outpatient care (n = 35), ambulatory care (n = 5), primary care (n = 20), community hospitals (n = 6), and pharmacies (n = 5). The studies were performed in the following countries: the USA (n = 49), the Netherlands (n = 6), cross-country (n = 5), Australia (n = 4), the United Kingdom (n = 2), Finland (n = 2), Germany (n = 2), Spain (n = 1), Belgium (n = 1), Israel (n = 1), Italy (n = 1), Taiwan (n = 1), and Japan (n = 1). Table 4 shows the factors that affect CDSS alert appropriateness in supporting clinical workflow, and Table 4 highlights key thee-specific issues and recommendation.

Table 4.

Factors that affect CDSS alert appropriateness in supporting clinical workflow.

Technology	Human	Organization	Process
System design Alert overload (n = 14)Alert content (n = 9)Alert specificity/sensitivity (n = 9)Usability (n = 8)System implementation Communication (n = 5)	Workload (n = 11)Training, Knowledge, and Skills (n = 4)	Internal rules and security policies (n = 4)	Alert presentation (n = 11)Timing of alerts (n = 8)Delay (n = 4)OutcomesPhysician productivity (n = 6)Appropriate alerts (n = 3)

Technology

Fourteen studies reported that an overwhelming number of inappropriate DDI and drug-allergy alerts are received through different communication channels during prescription, resulting in increased override rates (Table 5).^{6,7,38–46,95,107} In a previous study, 2700 alerts in an ambulatory care were triggered to prevent a single adverse drug event, and 90% of the alerts were overridden.⁷ Five studies reported that alert overload increases the mental workload of clinicians and decreases situation awareness in the workplace, causing clinicians to miss important information.^2,6,7,45,107 Although alert overload can overwhelm clinicians, limited information can also pose serious harm to patients.^43,44,46 Hence, alerts should be configured to provide relevant and sufficient information to its users without overwhelming them with unnecessary alerts.^7,42

Table 5.

Key themes, specific issues, and recommendations.

Socio-technical dimension	Five rights principle³³	Key theme	Specific issue	Recommendation
Technology	Right channelRight information	Alert overloadAlert content	Excess information^47,48 Desensitization from alert fatigue⁴¹ Non-specific patient information^49,50	Reducing the number of alerts that trigger during the patient visit will reduce the burden of clinicians.^41,51 Alert should be tiered according to severity (i.e. order of importance) to ensure that only important or relevant interactions or allergy alerts are presented to clinicians.^44,52
Human	Right person	Alert flexibility	Inability to control/ fine-tune alerts^48,49,53	Enabling and training users to tailor or control alerts will enhance clinician workflow.⁵⁴
Organization	Right policies	Deficient organizational rules/ regulations	Institutional rules/ government regulations impede alert use⁵⁵	Organizational policies should be well defined and flexible to support CDSS alert implementation.^55,56
Process	Right intervention formatRight time in the workflow	Alert presentationPoor alert timing	Interruptive/non-interruptive alerts^4,44,53 Alerts were triggered after consultation^52,57	Interruptive alerts should be designed to only notify clinicians of high-risk situations.^4,44 Alerts should be designed to trigger as early as possible during prescription.^47,52,58

Nine studies conducted in primary care, inpatient, and outpatient settings indicated that stand-alone CDSS alerts do not provide clinicians with sufficient information needed to make prescribing decisions.^3,49,59–67 Two surveys reported that the information displayed by drug alerts during medication order entry is difficult to interpret, unclear, and unconcise, thereby slowing down clinicians’ work.^3,59,60 Drug alerts are also redundant and display information that does not match the needs of individual or different users (e.g. general practitioner, pharmacist).^50,61,62 Another study carried out in outpatient setting showed that the quality of alert information, such as signal word (e.g. caution) and hazards (e.g. seriousness), determines whether or not the clinicians cancel a drug with DDI risk or continue with the prescription.^63,65 Clinicians emphasized that drug alerts should be concise, informative (about the patient’s condition, i.e. diagnoses, allergies, interactions, and medical history), and provide options for alternative medications.^3,60,62,65 Clinicians also claimed that having useful information at the point of care saves them from going through the tedious tasks of searching the system for additional information (e.g. vitals, dosage, weight, and height).^60,61

Nine studies conducted in inpatient and outpatient settings reported that poor specificity and sensitivity of CDSS alerts are the major reasons for alert overload.^{5,13,45,52,68–73} The CDSS alerts support drug prescription, administration, and are integrated into CPOE or EHR systems. Five studies showed that alerts with high specificity and low sensitivity generate a high number of inappropriate alerts, which result in alert overload.^45,68–71 In a pediatric center, nearly two-thirds of drug dose alerts are overridden, raising questions about alert specificity and the accuracy of the displayed information.⁶⁸ In this regard, clinicians emphasized that alerts should be highly specific, sensitive, and contain warnings indicating the severity level (e.g. critical, danger, or high risk).^52,68,71,72 Three studies suggested that alert specificity/sensitivity can be improved by modifying alerting rules in the knowledge base, default settings, codes for specific patient groups, and tailoring drug safety alerts to the clinical context.^5,13,52 In one outpatient setting, alert specificity and sensitivity are increased by re-categorizing the knowledge base of the DDI alerts and suppressing inappropriate alerts that are overridden by clinicians.⁵

Eight studies conducted in primary care,^{47,48,50,55,57} outpatient setting,⁷⁴ and community hospitals⁷⁵ reported that alerts integrated into CPOE or EHR systems cause clinicians to experience difficulties in distinguishing between serious and less serious DDI, drug duplication, and drug allergy alerts during prescription because of poorly designed interface.^{47,48,50,55,57,74–76} Some of these studies showed that poorly designed alert attributes, such as font, icon, label, and drop-down menu, lead to wrong order selection and juxtaposition errors.^47,48,50 Clinicians have to scroll through medication lists that are counter-intuitively arranged.^47,48,50 Medication lists grouped with similar drug names increase the chances of clinicians selecting the wrong drugs.^50,76 Clinicians prefer the basic principle of user interface design, such as minimized scrolling, mouse clicks, and steps, to reduce their chances of prescribing wrongly.^48,57,74 Five studies conducted in inpatient pharmacies reported that communication problems persist because clinicians face difficulty in understanding and interpreting alert messages that prevent them from using the alerting functions optimally.^61,77–80 Clinicians often ask for help, such as calling a pharmacist, to clarify why an alert is triggered and request to a have real-time or face-to-face communication with a pharmacist to interpret and resolve an alert that may endanger patients.^77–79

Human

Eleven studies conducted in primary care and ambulatory care settings showed that asynchronous alerts embedded in EHR increase the workload of clinicians. The asynchronous alerts inform clinicians about test results and medication refill requests via email, resulting in alert-related workload.^41,43,81 On average, clinicians in a primary care setting receive over 56 alerts per day and spend 49 min responding to asynchronous alerts.⁴² These alerts burden clinicians in terms of physical fatigue and cognitive weariness.^81,82 Providing clinicians with protected time to respond to alerts reduces alert-related workload and improves patient safety.^43,81,82

Other instances of workload involve clinicians documenting clinical data into EHR (e.g. dose route, frequency, and duration) before an alert can be triggered and selecting reasons for bypassing medication alerts during order entry.^{52,54,57,82–85} Workload of clinicians can be reduced by minimizing data entry tasks, decreasing the overall number of reminders, and eliminating work duplication.⁵⁴ CDSS alerts should be configured according to the respective user. For instance, an alert notifies a nurse when a patient shows early signs of sepsis, enabling a clinician to place an order using a sepsis order set.⁸⁶ Clinicians recommend options to accept or decline important alerts that may pose a threat to patient or normal alerts that provide known information.^54,57

Four studies conducted in primary care and inpatient pharmacies reported inadequate training, knowledge, and skills as a barrier to effective use of CPOE alerts during medication order entry.^49,50,54,78 Clinicians could not view, remove inapplicable, or activate reminders because of insufficient training, which lead to difficulty in managing alerts.^49,54 A study found that the lack of knowledge of providers on how to use CPOE for reviewing and verifying medication orders increases pharmacist workload.⁷⁸ Another study showed that educating clinicians through e-mail or monthly clinic meetings is ineffective because e-mails are often unread, and meetings are missed because of the heavy workload of clinicians.⁵⁰ These clinicians prefer hands-on training provided by local physicians and pharmacists.⁵⁰

Organization

Three studies in primary care and in outpatient setting showed that deficiencies in organizational rules and security policies impede alert use, contribute to HIT failure, and disrupt workflow.^36,59,87 According to Zazove et al.,⁵⁵ institutional rules prohibit clinicians from modifying the best practice alert (BPA) fonts, colors, and word placement to suit their personal needs. Another study reported that alerts display information that conflicts with the hospital policy and standard medication practices.⁶⁴ Furthermore, security policies or measures such as providing justification to override password before prescription completion and enforcing role-based access control for clinicians hinder the effective use of CDSS alert.⁸⁷

Process

Using the Lean method, we identified various wastes that affect CDSS appropriateness (Table 6). Clinician’s preferences regarding alert presentation format (i.e. interruptive or non-interruptive) during prescription and medication order entry varied across 11 studies.^{4,48,53,88–95} Interruptive or active alerts require clinicians to respond to the alerts before proceeding with other tasks, whereas non-interruptive alerts are passive alerts that clinicians choose to review.^4,48,91 Scheepers et al.⁴ compared four alert presentation methods (i.e. pharmacy intervention, physician alert list, EHR section, and pop-up alerts) to determine the clinician’s preferred method of alerting. The findings showed that active alerts, such as pop-ups, are more effective than passive alerts because they automatically appear during the workflow, reminding clinicians of items they might have overlooked.⁴

Table 6.

Waste in Lean IT and CDSS alert.

Waste type	Waste in Lean IT	Illustration in CDSS
Overproduction	Producing excessive information that is not needed or useful	Overwhelming number of inappropriate DDI and drug-allergy alerts (i.e. alert fatigue) that are received during prescription increases override rate.^2,6,7,45
Transportation/Motion	Extra steps or unnecessary repeated activity taken to accommodate inefficient processes	Clinicians scroll through medication lists that are counter-intuitively arranged to access needed information.^47,48,50
		Searching EHR for additional information (e.g. vitals, dosage, weight, and height).^47,48,50
		Clinicians call a pharmacist to query why a drug combination triggers an alert.⁷⁷
Inventory	Information waiting to be processed (e.g. excessive backlog and work in process)	Clinicians need to enter data in EHR (e.g. dose route, frequency, and duration) before an alert can be triggered.^52,54,57,83
Unrealized human potential	Unused or underutilized people talents caused by inadequate training and skills	Clinicians could not view reminders or remove inapplicable reminders due to lack of skills.^49,54
Over processing	Redundant processing of tasks that do not add value to users.	Clinicians must justify reasons for overriding a medication alert.
Over processing		Entering an override password before prescription can be completed.^52,54,57,83
Waiting (delays)	Unnecessary delays caused by software malfunctions (e.g. poor computer response time) and system downtime	Clinicians must wait for an alert before proceeding to other tasks because of slow processing time, computer crashes, and Internet connectivity problems.^96,97
Waiting (delays)		CPOE did not provide clinicians with sufficient time to enter and verify orders because of wrongly configured default settings.⁷⁹
Defects	Software features and functions that do not conform to the needs of users	Poorly designed alert font, icon, label, and drop-down menu lead to wrong order selection and juxtaposition errors.^47,48,50
Defects		Information displayed by drug alerts during medication order entry is difficult to interpret, unclear, and unconcise.^3,60

Conversely, other clinicians indicated that they prefer alerts to be displayed in a non-interruptive manner because they can be accessed at any time.^19,46 A survey conducted by Kucher et al.⁹² showed that the use of non-interruptive alerts increases clinician’s compliance with care measures by preventing venous thromboembolism (VTE) prophylaxis among hospitalized patients. These studies have shown that careful consideration is needed to determine alerts that should be designed as either interruptive or non-interruptive.^4,48,53,92 Phansalkar et al.⁹³ used a pragmatic approach to identify 33 low-priority DDI alerts that can be safely presented in an interruptive manner during workflow. Alert tiering by risk severity has been suggested as a strategy for presenting urgent and less urgent information to clinicians.^88,89,94,95 For example, level 1 alerts (drug contraindication) indicate a very serious interaction that requires clinicians to discontinue the drugs or cancel an order. Level 2 alerts (moderate or severe risk) refer to a less serious DDI but also require clinicians to provide valid reasons before completing a drug prescription. Level 3 alerts (mild or moderate risk) are the least serious interactions that appear in a non-intrusive manner at the bottom of the computer screen.⁹⁰

Seven studies reported that drug allergy and dosing alerts triggered after a medication are prescribed or dispensed.^{53,59,78,98–100} With CPOE use, determining the appropriate time for administering medication becomes difficult for clinicians because of wrongly configured default timing settings.⁷⁸ Another study⁹⁹ found that an inappropriate medication timing or medication timing error occurs when medication doses are not administered according to the standard administration time. These studies showed that pharmacists must use their own experience and judgment to ascertain the appropriate timing for administering medication.^78,99 Whether or not alerts should be triggered at different times is unclear, such as (1) when reviewing medication, (2) entering patient information, (3) when selecting an order, or (4) after an ordering session has been completed.^58,91,98 Therefore, the appropriate timing for alerting user needs to be determined to improve their effectiveness and increase their use.^51,101

Five studies identified unnecessary delays during clinical care processes in inpatient and outpatient clinics that were caused by hardware and software malfunctions.^8,56,96–98 Examples include clinicians waiting for an alert before proceeding to other tasks because of slow processing time, computer crashes, and Internet connectivity problems.^96,97 Saleem et al.⁵⁷ found that when the computer system crashes, clinicians could do little until the system is resumed because all nursing and provider functions are performed through the computerized provider record system (CPRS). Similarly, McMullen et al.⁷⁸ found that the safety features in a CPOE delay medication ordering. The study further revealed that clinicians experience system lockout every 10 min because the CPOE does not provide sufficient time to enter and verify orders.⁷⁸ Hence, hardware and software should be properly configured so that alerts can be activated in a timely manner.⁸

Outcomes

Researchers are yet to conclude whether or not the use of CDSS alert has improved clinical outcomes. Six studies conducted in ambulatory care, inpatient, and outpatient settings showed that poorly designed CDSS alerts reduce physician productivity and consequently affect healthcare quality and clinical outcomes.^{3,38,42,45,86,99} For instance, usability problems such as the number of clicks required to prescribe a drug or create referrals obscure the productivity of clinicians. The use of CDSS alerts and reminders in various care settings and clinical specialties to support care processes during personal interaction with patients contributes to productivity loss.^{3,38,42,45,86,99} Therefore, CDSS alerts must be designed to provide clinicians with patient-specific assessment and recommendation (i.e. appropriate alert) during task execution to improve the quality of care and clinical outcomes.^3,44,49

Discussion

This systematic review identifies factors affecting CDSS alert appropriateness in supporting clinical workflow from a socio-technical perspective, five rights principles, and Lean. Clinician’s response to medication alerts decreases as the total number of inappropriate alerts increases.^25,94 Managing alert overload burdens healthcare institutions because it requires a careful analysis of institution-specific practice and alert data.^103,105 Alert overload can be addressed by using data from reported alert usage, such as alert type (e.g. interaction, allergy, and duplicate drug), trigger (e.g. medication, dose, route, and schedule), result (e.g. alert override and order abandonment), and reason for overrides, to suppress inappropriate alerts for specific users.¹⁰⁵ Previous studies suggested many strategies to counteract alert overload, such as turning off frequently overridden alerts,¹⁰⁴ prioritizing alerts,⁴² tiering alert by severity,^52,104 and implementing highly specific algorithms.⁶⁹ Increasing the sensitivity and specificity of CDSS alerts in different care settings can positively influence system use and ultimately increase alert acceptance.^32,52 The specificity of a CDSS is a measure of its ability to distinguish between events that put an individual at risk of harm and non-events that will not: the more false positives, the lower the specificity.⁵² Sensitivity is the ability of the system to alert prescribers correctly when patients are at risk of experiencing drug-induced harm, such as from a DDI or drug allergy.⁵² Alert overload also occurs when knowledge base rules are wrongly edited or configured.¹⁰⁶ A complex system such as CPOE is usually integrated in one or more knowledge bases of drug dosing rules for reference purposes.^53,103 Therefore, knowledge bases for CDSS should be designed to display alerts judiciously by maintaining the right balance between inappropriate and relevant or useful alerts with important interactions that can undermine patient safety.^53,105

Alert appropriateness is also attributed to information content and display. Although incomplete or outdated allergy information in EHR leads to the proliferation of inappropriate alerts, incomplete allergy documentation still occurs because clinicians lack understanding on how to document information and what information needs to be documented (e.g. self-reported allergy, confirmed allergy, and nature of the reaction).^105,107 This scenario results in fragmented allergy documentations in patient records and subsequent delays in treatment.¹⁰⁷ To facilitate appropriate alert triggers, an updated and accurate allergy information must be documented and captured in standard location within the EHR.^33,108 Additional research is needed to ascertain how to best record this information and present clinicians with useful drug alerts.¹⁰⁷ Alerts should be designed or configured to provide sufficient information (e.g. frequency of interaction occurrences and alternative medications) that can enhance the decision making of clinicians.

Few studies focused on how alerts should be designed and display information to support the workflow of clinicians.^25,26 Careful design consideration of when to present alerts to prescribers or where they are displayed in the workflow is crucial. The conventional method of presenting information in the user interface is via interruptive and non-interruptive model dialogs (e.g. pop-up alerts).^16,25,109 Interruptive alerts are effective in obtaining the attention of clinicians, but these alerts have several usability problems, such as scrolling to view information and missing information at the point of care.^16,109 Presenting alerts based on its severity effect (i.e. the order of importance) allows prescribers to quickly identify important alerts and subsequently reduce alert fatigue and override.²⁵ Alert timing and its execution point are key factors in ensuring clinicians response.¹¹⁰ Clinicians have different preferences regarding alert timing across different care settings.^111,112 Some clinicians in inpatient and outpatient settings prefer receiving an interaction alert early in the decision-making process, particularly when selecting medication.¹¹¹ However, clinicians in primary care settings prefer viewing or receiving an alert at the end of the patient visit.¹¹²

Overall, our review findings across care settings and clinical specialties show that alerts are frequently ignored because they are inconsequential, too frequent, ineffective in design, implementation, and poor fit with workflow, thereby violating the five rights of CDSS. Other contributing factors include rigid organization structure and policies, and legal and ethical issues.^103,113 For instance, EHR developers are reluctant to remove alerts because of the fear of being held accountable if patients are harmed in the absence of a warning.¹¹³ According to Reidmann et al.⁷³, the five rights may be better achieved when alerts are tailored and filtered to consider the characteristics of (1) an organization and its users and (2) a patient or a case. Thus, socio-technical factors, such as workload, communication, usability, and organizational policies, are crucial for the appropriate design and implementation of CDSS alerts. Insufficient knowledge and skills regarding basic CPOE function (e.g. alerts) hinder the effective use of CDSS alerts. Education and training should provide clinicians with an understanding of the benefits of CDSS and how clinicians can respond appropriately to alerts.¹⁰⁵ Such knowledge would help clinicians have a better understanding on how CDSSs function.¹⁰⁵

Care settings and clinical specialties play an important role in managing alert. Presenting the same drug alerts to clinicians in different care settings and clinical specialties may contribute to inappropriate alert overrides. For instance, increase in override rates for various alert types, such as duplicate drugs, DDI, drug allergy, and formulary substitution, ranges from 50% to 90% in inpatients and 33% to 90% in outpatients.^114–116 These overrides are appropriate when patients had previously tolerated the medication and the provider would monitor for the recommended DDI and alert duplication.^114–116 In primary care settings, reasons for overriding alert include lack of relevance to specific clinical situation, lack of timeliness, and “benefit outweigh risk.”^101,107 Therefore, fine tuning drug alerts to match the needs of clinicians in different care settings and clinical specialties may improve the response of clinicians to alerts.^2,107

Analysis of CDSS inefficiencies using the waste category revealed the specific limitation of CDSS in supporting clinical workflow. Clinicians spend time performing NVA activities, such as data entry task, switching between screens, and logging into multiple systems. Thus, standardized process improvement methods, such as Lean and BPM, can be used to gain a deep understanding of care processes performed by clinicians, such as the information they use and user–computer interaction. Process analysis can identify process efficiency, problems, and automation potential to improve the process. These detailed insights may inform the design of CDSS alerts. The combination of approaches, namely, socio-technical model, five rights of CDSS, and Lean, can be adopted to provide a holistic view in analyzing, improving, and redesigning CDSS alerts. Our socio-technical approach guides us in interpreting the review findings. However, some measures in the framework could be categorized under multiple dimensions. For example, within the process dimension, measures such as alert timing and alert presentation may also be categorized under the technology dimension. Further application of the proposed evaluation dimensions and measures may lead to additional or specific new measures.

A systematic review ideally involves a group of reviewers to collaboratively implement several steps to avoid bias in the review process. The steps include paper screening, data extraction, and evaluation of study quality. Our review involves researcher OO, a PhD candidate, as a sole reviewer; but this limitation is addressed through a series of consultations between researcher OO and supervisor MMY, who has experienced in conducting systematic review. The discussions include designing the overall study and systematic review method; developing a protocol on the basis of the PRISMA method; adhering to the protocol; defining clear inclusion and exclusion criteria for the referral of different paper screening stages; designing and mapping the data extraction form and extracting data on the basis of the study objectives; and assessing the quality of the selected studies. In addition, the review process is documented in detail to enable readers assess the thoroughness of the study.¹¹⁷ However, we realized that as an untrained reviewer, the conduct of the review process, including the criteria assessment, is prone to error,¹¹⁸ subjectivity, bias, and lack of rigor, all of which may affect the generalizability and applicability of the findings.^119,120

Conclusion

CDSSs can improve patient safety and quality of care, but poorly designed CDSSs may undermine patient safety. This review shows that human factor methods and principles have not been properly applied to CDSS design. Our review identified factors affecting CDSS alert appropriateness in supporting clinical workflow, including technology (alert content, usability, information overload), human (alert flexibility, workload and training), organization (internal rules and policies), and process (alert presentation, alert timing, delays). These factors can inform practitioners on how a CDSS alert can be designed to fit into clinical workflow. Future research should focus on how human factor engineering principles, socio-technical framework, and process improvement methods, such as BPM and Lean, can be used to inform the design of CDSS alerts.

Footnotes

Appendix

Appendix A

Characteristics of selected studies.

Study (author)	CDSS tool type	Design	Country	Settings	Number of sites	Study target	Types of alerts
Kassakian et al.¹³	Integrated in EHR	Retrospective cohort study	USA	Inpatient	Single	Clinicians	Medication alerts
Ancker et al.⁸³	Integrated in EHR	Retrospective cohort study	USA	Primary care	Single	Clinicians	Drug and clinical reminders
Austrian et al.⁸⁷	EHR	Retrospective cohort study	USA	Primary care	Single	Clinicians	Sepsis alert
Seidling et al.⁷⁰	Standalone CDSS	Retrospective cohort study	Germany	Outpatient	Single	Clinicians	DDI alerts
Cuéllar Monreal et al. (2017)	AEPS	Mixed method	Spain	Inpatient	Single	Clinicians	DDI and allergy alerts
Baseman et al.⁴²	e-prescribing	Qualitative study	USA	Public health agencies	Single	Health care professionals (HCP)	E-mail notifications
Murphy et al.⁴⁴	Integrated in EHR	Retrospective study	USA	Primary care	Single	Primary care practitioners (PCP)	Asynchronous alerts
Wright et al.⁸	Integrated in EHR	Mixed method	USA	Outpatient settings	Single	Chief medical informatics officers, CDS developers	DDI alerts
Bauer et al.⁷⁶	Standalone CDSS	Retrospective cohort study	USA	Not stated	Single	Clinicians and patients	CHICA Prompts
Carroll et al.⁹⁶	Standalone CDSS	Retrospective cohort study	USA	Not stated	Single	Clinicians	CHICA Prompts
McEvoy et al.⁹⁵	Integrated in EHR	Retrospective study	USA	Inpatient and outpatient	Multi	Clinicians	Dosing alerts
Stultz et al.⁶⁹	CDSS	Retrospective study	USA	Inpatient	Single	Clinicians	Dosing alert
Van der Sijs, et al.⁷²	Integrated in CDSS	Retrospective study	Netherland	Inpatient	Single	Clinicians	TTDI alerts
Blecker et al.⁵⁴	CDSS	Qualitative	USA	Inpatient	Single	Clinicians	ACE alerts
Jani et al.⁷⁸	e-prescribing	Retrospective cohort study	UK	Tertiary care and Paediatric center	Single	Clinicians	Drug allergy
Chused et al.⁸⁰	CPOE with CDSS	Retrospective cohort study	USA	Not stated	Single	Clinicians	Drug alerts
Bryant et al.⁴⁶	CDSS	Retrospective cohort study	USA	Inpatient	Single	Clinicians	DDI alerts
Idemoto et al.¹⁰⁰	CPOE	Retrospective cohort study	USA	Primary care	Single	Clinicians	Drug alerts
Russ et al.⁶⁵	CPOE with CPRS	Observation (qualitative study)	USA	Outpatient	Single	Clinicians	DDI
Jaensch et al.¹⁰²	CPOE with CDSS	Qualitative study	Australia	Emergency department	Single	Junior doctors	DDI
Sweidan et al.⁶¹	CDSS	Comparative study	Australia	Primary care	Single	Clinicians	DDI
Seidling et al.⁶⁴	CDSS	Retrospective cohort study	USA	Inpatient	Single	Clinicians	DDI and drug duplicate
Nasuhara et al.⁸⁸	DAIS	Retrospective cohort study	Japan	Outpatient	Single	Physicians	DDI and drug duplicate
Smithburger et al.⁸⁹	CDSS	Expert opinion	USA	Not stated	Single	Clinicians	DDI
Fischer et al.⁹⁰	e-prescribing	Retrospective cohort study	USA	Inpatient	Single	Clinicians	Drug formulary
Patterno et al.⁹¹	CPOE	Retrospective cohort study	USA	Inpatient	Multi	Adult inpatients and providers	DDI
Trinkley et al.⁹²	CDS	Qualitative (focus group)	USA	Primary care	Single	Clinicians	Drug allergy
McDaniel et al.¹⁰²	CDSS with CPOE	Retrospective cohort study	USA	Inpatient	Single	Clinicians	DDI
Zenziper et al.⁷¹	CDSS	Retrospective cohort study	Israel	Multipledepartments	Single	Clinicians	DDI, drug duplicate, and dose
Weingart et al.¹⁰⁴	e- prescribing system	Quantitative (survey)	USA	Ambulatory care	Single	Clinicians	DDI
Ash et al. (2004)	PCIS with CDSS	Qualitative (interviews)	Cross country	Inpatient and Outpatient	Single	Pharmacists, Physician,Laboratory, and Medical records staff	Warning alert
Koskela et al.⁸⁴	CDSS	Qualitative (focus group)	Finland	Primary care	Single	Physicians and nurses	DDI and drug dosing
Birmingham et al.⁴⁸	Integrated in EHR	Qualitative (focus group, interview)	USA	Primary care	Single	Clinicians	Vaccination alerts
Coleman et al.⁵³	CDSS with CPOE	Expert Participant discussions	Cross country	Workshop	Not stated	Not stated	Medication alerts
Krall & Sittig⁴⁹	EMR	Qualitative study (focus group)	USA	Primary care	Single	Clinicians	Drug alerts and reminders
Feldstein et al.⁵¹	Integrated in CPOE	Qualitative study (interviews)	USA	Primary care	Single	Prescribers	Drug safety alerts
Saleem et al.⁵⁸	Integrated in EMR	Qualitative study (observation, interviews)	USA	Outpatient	Multi	Clinicians and nurses	Clinical reminders
Lugtenberg et al.⁵⁰	CDSS	Qualitative study (focus group)	Netherlands	Primary care	Single	Primary care practitioners (PCP)	Drug alerts
Dixon et al.⁶⁷	CDSS	Qualitative study (interviews)	USA	Outpatient	Multi	Clinicians and nurses	HPV reminders
Baysari et al.⁵⁹	CDSS with CPOE	Qualitative study (observations, interviews)	Australia	Ward rounds	Single	Clinicians	Drug allergy and therapeutic duplication
Russ et al.⁶²	CPRS with CPOE	Qualitative study (observations, interviews)	USA	Primary care	Single	Prescribers	Drug alerts
Zazove et al.⁵⁶	EHR	Qualitative study (interviews)	USA	Primary care	Multi	Clinicians	Best practice alerts (BPA)
Trivedi et al.⁹⁸	CDSS	Observations	USA	Outpatient	Multi	Clinicians and nurses	Reminders for guidelines
Kucher et al.⁹³	Electronic prescribing system	Retrospective cohort study	Switzerland	Outpatient	Single	Clinicians	VTE alerts
Stone⁶⁶	Integrated in EHR	Case study	USA	Inpatient and outpatient	Multi	Clinicians	Drug safety alerts
Embi and Leonard⁵²	EHR	Randomized cohort study	USA	Not stated	Single	Clinicians	Clinical trial alerts
Phansalkar et al.⁹⁴	EHR	Retrospective cohort study	USA	Outpatient	Single	Clinical informaticists and clinicians	DDI alerts
Cornu et al.⁴⁴	CDSS	Quantitative (survey)	Belgium	Inpatient	Single	Clinicians	DDI and drug allergy
St Sauver et al.⁶³	CDSS	Quantitative (survey)	USA	Primary care	Single	Clinicians	PGx-CDS alerts
Long et al.³	CDSS	Quantitative (survey)	USA	General practice	Single	Nurses	Sepsis alert
Glassman et al.⁴⁰	CPRS	Quantitative (survey)	USA	Inpatient and Outpatient	Single	Clinicians	DDI alert
Magnus et al.⁴¹	EMIS	Quantitative (survey)	United Kingdom	Primary care	Multi	General practitioners (GP)	Drug interaction alerts
Jung et al.⁴³	CPOE with CDSS	Quantitative (survey)	Cross country	Cross hospitals	Multi	Physicians	DDI and dosing alerts
Campmans et al.⁷⁷	PIS	Quantitative (survey)	Netherlands	Community pharmacies	Single	Pharmacists and pharmacy technicians	Drug name and drug confusion alert
Scheepers et al.⁴	CDSS	Retrospective cohort study	Netherlands	Inpatient	Clinicians	Single	Pop-up, pharmacy interventions, physician alert list
Patterson et al.⁵⁵	CPRS	Mixed method	USA	Outpatient	Clinicians	Single	HIV clinical reminders
Lapane et al.⁶	e-prescribing	Mixed method	USA	Primary care	Clinicians	Multi	DDI
Gregory et al.⁸²	EHR with CPRS	Mixed methods	USA	Primary care	Clinicians	Single	Asynchronous alerts
McMullen et al.⁷⁹	CPOE	Mixed method	USA	Inpatient pharmacies	Pharmacist	Multi	Drug allergy
Hayward et al.⁹⁹	CDSS	Mixed method	UK	Primary care	Single	General practitioners and patients	DDI
Stultz and Nahata⁹	CDSS	Retrospective analysis	USA	Inpatient	Single	Patients	Dosing alerts
Weingart et al.¹⁰³	e-prescribing	Retrospective cohort study	USA	Ambulatory care	Single	Clinicians	Medication safety alerts
Saiyed et al.³⁹	CPOE	Retrospective cohort study	USA	outpatient	Single	Clinicians	Drug dose alerts
Hsieh et al.⁴⁷	Integrated in CPOE	Retrospective cohort study	USA	Inpatient	Single	Clinicians	Drug allergy alerts
Varonen et al.⁸⁵	CDSS	Qualitative study (focus group)	Finland	Outpatient	Multi	Clinicians	DDI
Schnipper et al.⁸⁶	Integrated in EHR	Mixed method (survey and interviews)	USA	Ambulatory care	Single	Clinicians	Smart forms and reminders
Noblin et al.⁸¹	EHR	Qualitative study (interviews)	USA	Not stated	Single	Clinicians	Medication safety alerts
Ash et al.⁶⁸	CDSS	Qualitative study (interviews)	USA	Inpatient and outpatient	Single	Clinicians	DDI and drug allergy
Saxena et al.⁷³	Integrated in CPOE	Retrospective cohort study	USA	Outpatient	Single	Clinicians	Drug duplication and contradictions
Steele et al.⁶⁰	Integrated in CPOE	Retrospective cohort study	USA	Primary care	Single	Clinicians	Drug laboratory interactions

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: We gratefully acknowledge the funding received from the Universiti Kebangsaan Malaysia (DIP 2016-033) and Ministry of Higher Learning Malaysia (FRGS/1/2018/ICT04/UKM/02/5).

ORCID iD

Maryati Mohd Yusof

References

IOM. Health IT and patient safety. Washington, DC: National Academies Press, 2012.

Fant

Adelman

Too many medication alerts. Nurse Pract 2018; 43(11): 48–52.

Long

Capan

Mascioli

, et al. Evaluation of user-interface alert displays for clinical decision support systems for sepsis. Crit Care Nurse 2018; 38(4): 46–54.

Scheepers-Hoeks

A-MJ

Grouls

Neef

, et al. Physicians’ responses to clinical decision support on an intensive care unit—Comparison of four different alerting methods. Artif Intell Med 2013; 59(1): 33–38.

Wright

Hickman

T-TT

McEvoy

, et al. Analysis of clinical decision support system malfunctions: a case series and survey. J Am Med Informatics Assoc 2016; 23(6): 1068–1076.

Lapane

Waring

Schneider

, et al. A mixed method study of the merits of e-prescribing drug alerts in primary care. J Gen Intern Med 2008; 23(4): 442–446.

Weingart

Simchowitz

Padolsky

, et al. An empirical model to estimate the potential impact of medication safety alerts on patient safety, health care utilization, and cost in ambulatory care. Arch Intern Med 2009; 169(16): 1465.

Wright

Ash

, et al. Clinical decision support alert malfunctions: analysis and empirically derived taxonomy. J Am Med Inform Assoc 2018; 25(5): 496–506.

Stultz

Nahata

MC.

Appropriateness of commercially available and partially customized medication dosing alerts among pediatric patients. J Am Med Informatics Assoc 2014; 21(e1): e35–e42.

10.

McCoy

Thomas

Krousel-Wood

, et al. Clinical decision support alert appropriateness: a review and proposal for improvement. Ochsner J 2014; 14(2): 195–202.

11.

Olakotan

Yusof

MM.

Evaluating the alert appropriateness of clinical decision support systems in supporting clinical workflow. J Biomed Inform 2020; 106: 103453.

12.

Olufisayo

Mohd Yusof

Ezat Wan Puteh

Enhancing CDSS alert appropriateness in clinical workflow using the lean method. Stud Health Technol Inform 2018; 255: 112–116.

13.

Kassakian

Yackel

Gorman

, et al. Clinical decisions support malfunc-tions in a commercial electronic health record. Appl Clin Inf 2017; 8: 910–923.

14.

Osheroff

Improving medication use and outcomes with clinical decision support: a step by step guide. Published online 2009.

15.

Bowman

Impact of electronic health record systems on information integrity: quality and safety implications. Perspect Heal Inf Manag 2013; 10(Fall): 1c.

16.

Horsky

Phansalkar

Desai

, et al. Design of decision support interventions for medication prescribing. Int J Med Inform 2013; 82(6): 492–503.

17.

Sittig

Wright

Osheroff

, et al. Grand challenges in clinical decision support. J Biomed Inform 2008; 41(2): 387–392.

18.

Wasylewicz

ATM

Scheepers-Hoeks

AMJW

. Clinical decision support systems. In: Kubben P, Dumontier M, Dekker A (eds) Fundamentals of clinical data science. Springer International Publishing, 2019, pp.153–169.

19.

Yusof

Kuljis

Papazafeiropoulou

, et al. An evaluation framework for Health Information Systems: human, organization and technology-fit factors (HOT-fit). Int J Med Inform 2008; 77(6): 386–398.

20.

Parasuraman

Sheridan

Wickens

CD.

A model for types and levels of human interaction with automation. IEEE Trans Syst Man, Cybern - Part A Syst Humans 2000; 30(3): 286–297.

21.

Dumas

La Rosa

Mendling

, et al. Fundamentals of Business Process Management. Berlin, Heidelberg: Springer, 2018.

22.

Carayon

Schoofs Hundt

Karsh

B-T

, et al. Work system design for patient safety: the SEIPS model. Qual Saf Health Care 2006; 15 Suppl 1(Suppl 1): i50–i58.

23.

Dort

Baysari

MT.

Prescriber perceptions of medication-related computerized decision support systems in hospitals : a synthesis of qualitative research. Int J Med Inform 2019; 129(February): 285–295.

24.

Holden

Carayon

Gurses

, et al. SEIPS 2.0: a human factors framework for studying and improving the work of healthcare professionals and patients. Ergonomics 2014; 56(11): 1669–1686.

25.

Marcilly

Ammenwerth

Vasseur

, et al. Usability flaws of medication-related alerting functions: a systematic qualitative review. J Biomed Inform. 2015; 55: 260–271.

26.

Miller

Moon

Anders

, et al. Integrating computerized clinical decision support systems into clinical work: A meta-synthesis of qualitative research. Int J Med Inform 2015; 84(12): 1009–1018.

27.

Buttigieg

Dey

Gauci

Business process management in health care: current challenges and future prospects. Innov Entrep Heal 2016; 3: 1.

28.

Awang Kalong

Yusof

. Waste in health information systems: a systematic review. Int J Health Care Qual Assur 2017; 30(4): 341–357.

29.

Yusof

Khodambashi

Mokhtar

AM.

Evaluation of the clinical process in a critical care information system using the Lean method: a case study. BMC Med Inform Decis Mak 2012; 12(1): 150.

30.

Ker

Wang

Hajli

, et al. Deploying lean in healthcare: Evaluating information technology effectiveness in U.S. hospital pharmacies. Int J Inf Manage 2014; 34(4): 556–560.

31.

Berwick

Hackbarth

AD.

Eliminating waste in US health care. JAMA 2012; 307(14): 1513–1516.

32.

Martin

Eliminating waste in healthcare strategies to improve outcomes and reduce costs. ASQ Healthc Updat 2014; 1:7

33.

Kuperman

Bobb

Payne

, et al. Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc 2007; 14(1): 29–40.

34.

Osheroff

Improving outcomes with clinical decision support. HIMSS Publishing, 2012. https://www.routledge.com/Improving-Outcomes-with-Clinical-Decision-Support-An-Implementers-Guide/Osheroff-Teich-Levick-Saldana-Velasco-Sittig-Rogers-Jenders/p/book/9780984457731

35.

Borowski

Görges

Fried

, et al. Medical device alarms. Biomed Tech (Berl) 2011; 56(2): 73–83.

36.

Critical Appraisal Skills Programme. CASP critical appraisal skills programme. CASP Qual Checkl 2018; 2018: 1.

37.

Hong

QN.

mixedmethodsappraisaltoolpublic [licensed for non-commercial use only] / FrontPage. http://mixedmethodsappraisaltoolpublic.pbworks.com/w/page/24607821/FrontPage (2020, accessed 19 February 2021).

38.

Saiyed

Greco

Fernandes

, et al. Optimizing drug-dose alerts using commercial software throughout an integrated health care system. J Am Med Informatics Assoc 2017; 24(6): 1149–1154.

39.

Glassman

Simon

Belperio

, et al. Improving recognition of drug interactions: benefits and barriers to using automated drug alerts. Med Care 2002; 40(12): 1161–1171.

40.

Magnus

Rodgers

Avery

GPs’ views on computerized drug interaction alerts: questionnaire survey. J Clin Pharm 2002; 27(5): 377–382.

41.

Baseman

Revere

Painter

, et al. Public health communications and alert fatigue. BMC Health Serv Res 2013; 13(1): 295.

42.

Jung

Riedmann

Hackl

, et al. Physicians’ perceptions on the usefulness of contextual information for prioritizing and presenting alerts in computerized physician order entry systems. BMC Med Inform Decis Mak 2012; 12(1): 111.

43.

Murphy

Reis

Sittig

, et al. Notifications received by primary care practitioners in electronic health records: a taxonomy and time analysis. Am J Med 2012; 125(2): 209.e1–209.e7.

44.

Cornu

Steurbaut

Beukeleer

, et al. Physician’s expectations regarding prescribing clinical decision support systems in a Belgian hospital. Acta Clin Belg 2014; 69(3): 157–164.

45.

Bryant

Fletcher

Payne

TH.

Drug interaction alert override rates in the meaningful use era. Appl Clin Inform 2014; 5(3): 802–813.

46.

Hsieh

Kuperman

Jaggi

, et al. Characteristics and consequences of drug allergy alert overrides in a computerized physician order entry system. J Am Med Informatics Assoc 2004; 11(6): 482–491.

47.

Birmingham

Catallozzi

Findley

, et al. FluAlert: a qualitative evaluation of providers’ desired characteristics and concerns regarding computerized influenza vaccination alerts. Prev Med (Baltim) 2011; 52(3-4): 274–277.

48.

Krall

Sittig

DF.

Clinician’s assessments of outpatient electronic medical record alert and reminder usability and usefulness requirements. Proc AMIA Symp 2002; 400–404.

49.

Lugtenberg

Weenink

Van Der Weijden

, et al. Implementation of multiple-domain covering computerized decision support systems in primary care: a focus group study on perceived barriers. BMC Med Inform Decis Mak 2015; 15(1): 82.

50.

Feldstein

Simon

Schneider

, et al. How to design computerized alerts to ensure safe prescribing practices. Jt Comm J Qual Saf 2004; 30(11): 602–613.

51.

Embi

Leonard

AC.

Evaluating alert fatigue over time to EHR-based clinical trial alerts: findings from a randomized controlled study. J Am Med Informatics Assoc 2012; 19(e1): e145–e148.

52.

Coleman

van der Sijs

Haefeli

, et al. On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop. BMC Med Inform Decis Mak 2013; 13(1): 111.

53.

Blecker

Pandya

Stork

, et al. Interruptive versus noninterruptive clinical decision support: usability study. JMIR Hum Factors 2019; 6(2): e12469.

54.

Patterson

Doebbeling

Fung

, et al. Identifying barriers to the effective use of clinical reminders: bootstrapping multiple methods. J Biomed Inform 2005; 38(3): 189–199.

55.

Zazove

McKee

Schleicher

, et al. To act or not to act: responses to electronic health record prompts by family medicine clinicians. J Am Med Inform Assoc 2017; 24(2): 275–280.

56.

Cornu

Phansalkar

Seger

, et al. High-priority and low-priority drug–drug interactions in different international electronic health record systems: a comparative study. Int J Med Inform 2018; 111: 165–171.

57.

Saleem

Patterson

Militello

, et al. Exploring barriers and facilitators to the use of computerized clinical reminders. J Am Med Informatics Assoc 2005; 12(4): 438–447.

58.

Baysari

Westbrook

Richardson

, et al. The influence of computerized decision support on prescribing during ward-rounds: are the decision-makers targeted? J Am Med Informatics Assoc 2011; 18(6): 754–759.

59.

Steele

Eisert

Witter

, et al. The effect of automated alerts on provider ordering behavior in an outpatient setting. PLoS Med 2005; 2(9): e255.

60.

Sweidan

Reeve

Brien

, et al. Quality of drug interaction alerts in prescribing and dispensing software. Med J Aust 2009; 190(5): 251–254.

61.

Russ

Zillich

McManus

, et al. Prescribers’ interactions with medication alerts at the point of prescribing: a multi-method, in situ investigation of the human–computer interaction. Int J Med Inform 2012; 81(4): 232–243.

62.

St Sauver

Bielinski

Olson

, et al. Integrating pharmacogenomics into clinical practice: promise vs reality. Am J Med 2016; 129(10): 1093–1099.e1.

63.

Seidling

Phansalkar

Seger

, et al. Factors influencing alert acceptance: a novel approach for predicting the success of clinical decision support. J Am Med Informatics Assoc 2011; 18(4): 479–484.

64.

Russ

Zillich

McManus

, et al. A human factors investigation of medication alerts: barriers to prescriber decision-making and clinical workflow. AMIA. Annu Symp Proc AMIA Symp 2009; 2009: 548–552.

65.

Stone

EG.

Unintended adverse consequences of a clinical decision support system: two cases. J Am Med Informatics Assoc 2018; 25(5): 564–567.

66.

Dixon

Kasting

Wilson

, et al. Health care providers’ perceptions of use and influence of clinical decision support reminders: qualitative study following a randomized trial to improve HPV vaccination rates. BMC Med Inform Decis Mak 2017; 17(1): 119.

67.

Ash

Sittig

Campbell

, et al. Some unintended consequences of clinical decision support systems. AMIA Annu Symp Proc 2007; 2007: 26–30.

68.

Stultz

Porter

Nahata

MC.

Sensitivity and specificity of dosing alerts for dosing errors among hospitalized pediatric patients. J Am Med Inform Assoc 2014; 21(e2): e219–e225.

69.

Seidling

Klein

Schaier

, et al. What, if all alerts were specific – estimating the potential impact on drug interaction alert burden. Int J Med Inform 2014; 83(4): 285–291.

70.

Zenziper Straichman

Kurnik

Matok

, et al. Prescriber response to computerized drug alerts for electronic prescriptions among hospitalized patients. Int J Med Inform 2017; 107: 70–75.

71.

van der Sijs

Mulder

van Gelder

, et al. Drug safety alert generation and overriding in a large Dutch university medical centre. Pharmacoepidemiol Drug Saf 2009; 18(10): 941–947.

72.

Saxena

Lung

Becker

JR.

Improving patient safety by modifying provider ordering behavior using alerts (CDSS) in CPOE system. AMIA Annu Symp Proc 2011; 2011: 1207–1216.

73.

Riedmann

Jung

Hackl

, et al. Development of a context model to prioritize drug safety alerts in CPOE systems. BMC Med Inform Decis Mak 2011; 11(1): 35.

74.

Holden

RJ.

What stands in the way of technology-mediated patient safety improvements? a study of facilitators and barriers to physicians’ use of electronic health records. J Patient Saf 2011; 7(4): 193–203.

75.

Bauer

Carroll

Saha

, et al. Experience with decision support system and comfort with topic predict clinicians’ responses to alerts and reminders. J Am Med Informatics Assoc 2016; 23(e1): e125–e130.

76.

Campmans

van Rhijn

Dull

, et al. Preventing dispensing errors by alerting for drug confusions in the pharmacy information system—a survey of users. PLoS One 2018; 13(5): e0197469.

77.

Jani

Barber

Wong

ICK

. Characteristics of clinical decision support alert overrides in an electronic prescribing system at a tertiary care paediatric hospital. Int J Pharm Pract 2011; 19(5): 363–366.

78.

McMullen

Macey

Pope

, et al. Effect of computerized prescriber order entry on pharmacy: Experience of one health system. Am J Heal Pharm 2015; 72(2): 133–142.

79.

Chused

Kuperman

Stetson

PD.

Alert override reasons: a failure to communicate. AMIA Annu Symp Proc 2008; 2008: 111–115.

80.

Noblin

Cortelyou-Ward

Cantiello

, et al. EHR implementation in a new clinic: a case study of clinician perceptions. J Med Syst 2013; 37(4): 9955.

81.

Gregory

Russo

Singh

Electronic health record alert-related workload as a predictor of burnout in primary care providers. Appl Clin Inform 2017; 8(3): 686–697.

82.

Ancker

Edwards

Nosal

, et al. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak 2017; 17(1): 36.

83.

Koskela

Sandström

Mäkinen

, et al. User perspectives on an electronic decision-support tool performing comprehensive medication reviews - a focus group study with physicians and nurses. BMC Med Inform Decis Mak 2015; 16(1): 6.

84.

Varonen

Kortteisto

Kaila

, et al. What may help or hinder the implementation of computerized decision support systems (CDSSs): a focus group study with physicians. Fam Pract 2008; 25(3): 162–167.

85.

Schnipper

Linder

Palchuk

, et al. Smart forms; in an electronic medical record: documentation-based clinical decision support to improve disease management. J Am Med Inform Assoc 2008; 15(4): 513–523.

86.

Austrian

Jamin

Doty

, et al. Impact of an emergency department electronic sepsis surveillance system on patient mortality and length of stay. J Am Med Inform Assoc 2018; 25(5): 523–529.

87.

Nasuhara

Sakushima

Endoh

, et al. Physicians’ responses to computerized drug interaction alerts with password overrides. BMC Med Inform Decis Mak 2015; 15(1): 74.

88.

Smithburger

Buckley

Bejian

, et al. A critical evaluation of clinical decision support for the detection of drug–drug interactions. Expert Opin Drug Saf 2011; 10(6): 871–882.

89.

Fischer

Vogeli

Stedman

, et al. Effect of electronic prescribing with formulary decision support on medication use and cost. Arch Intern Med 2008; 168(22): 2433–2439.

90.

Paterno

Maviglia

Gorman

, et al. Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc 2009; 16(1): 40–46.

91.

Trinkley

Blakeslee

Matlock

, et al. Clinician preferences for computerised clinical decision support for medications in primary care: a focus group study. BMJ Heal Care Inform 2019; 26(1):1–8.

92.

Kucher

Puck

Blaser

, et al. Physician compliance with advanced electronic alerts for preventing venous thromboembolism among hospitalized medical patients. J Thromb Haemost 2009; 7(8): 1291–1296.

93.

Phansalkar

van der Sijs

Tucker

, et al. Drug-drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc 2013; 20(3): 489–493.

94.

McEvoy

Sittig

Hickman

T-T

, et al. Variation in high-priority drug-drug interaction alerts across institutions and electronic health records. J Am Med Inform Assoc 2017; 24(2): 331–338.

95.

Carroll

Anand

Downs

SM.

Understanding why clinicians answer or ignore clinical decision support prompts. Appl Clin Inform 2012; 3(3): 309–317.

96.

Castro

Buczkowski

Hafner

JM.

The contribution of sociotechnical factors to health information technology-related sentinel events. Jt Comm J Qual patient Saf 2016; 42(2): 70–76.

97.

Trivedi

Daly

Kern

, et al. Barriers to implementation of a computerized decision support system for depression: an observational report on lessons learned in "real world" clinical settings. BMC Med Inform Decis Mak 2009; 9: 6.

98.

Hayward

Thomson

Milne

, et al. ‘Too much, too late’: mixed methods multi-channel video recording study of computerized decision support systems and GP prescribing. J Am Med Inform Assoc 2013; 20(e1): e76–e84.

99.

Idemoto

Williams

Ching

, et al. Implementation of a custom alert to prevent medication-timing errors associated with computerized prescriber order entry. Am J Heal Pharm 2015; 72(17): 1481–1488.

100.

McDaniel

Burlison

Baker

, et al. Alert dwell time: introduction of a measure to evaluate interruptive clinical decision support alerts. J Am Med Inform Assoc 2016; 23(e1): e138–e141.

101.

Jaensch

Baysari

Day

, et al. Junior doctors’ prescribing work after-hours and the impact of computerized decision support. Int J Med Inform 2013; 82(10): 980–986.

102.

Weingart

Simchowitz

Shiman

, et al. Clinicians’ assessments of electronic medication safety alerts in ambulatory care. Arch Intern Med 2009; 169(17): 1627–1632.

103.

Iii

JDM

Mallozzi

Perkins

, et al. Reducing alert burden in electronic health records : state of the art recommendations from four health systems. Appl Clin Inform 2020; 11(1): 1–12.

104.

van der Sijs

Aarts

van Gelder

, et al. Turning off frequently overridden drug alerts: limited opportunities for doing it safely. J Am Med Inform Assoc 2008; 15(4): 439–448.

105.

Horn

Gumpper

Hardy

, et al. Clinical decision support for drug–drug interactions: Improvement needed. Am J Heal Pharm 2013; 70(10): 905–909.

106.

Kuperman

Bobb

Payne

, et al. Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Informatics Assoc 2007; 14(1): 29–40.

107.

Tolley

Slight

Husband

, et al. Improving medication-related clinical decision support. Am J Heal Pharm 2018; 75(4): 239–246.

108.

Temel

Greer

Gallagher

, et al. Electronic prompt to improve outpatient code status documentation for patients with advanced lung cancer. J Clin Oncol 2013; 31(6): 710–715.

109.

Horsky

Schiff

Johnston

, et al. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. J Biomed Inform 2012; 45(6): 1202–1216.

110.

Bell

Garfield

Khosla

, et al. Mixed methods study of medication-related decision support alerts experienced during electronic prescribing for inpatients at an English hospital. Eur J Hosp Pharm 2019; 26(6): 318–322.

111.

Humphrey

Mirica

Ozonoff

, et al. Clinician perceptions of timing and presentation of drug-drug interaction alerts. Appl Clin Inform 2020; 11(3): 487–496.

112.

Payne

Hines

Chan

, et al. Recommendations to improve the usability of drug-drug interaction clinical decision support alerts. J Am Med Informatics Assoc 2015; 22(6): 1243–1250.

113.

Kesselheim

Cresswell

Phansalkar

, et al. Clinical decision support systems could be modified to reduce “alert fatigue” while still minimizing the risk of litigation. Health Aff 2011; 30(12): 2310–2317.

114.

Nanji

Seger

Slight

, et al. Medication-related clinical decision support alert overrides in inpatients. J Am Med Inform Assoc 2018; 25(5): 476–481.

115.

Slight

Beeler

Seger

, et al. A cross-sectional observational study of high override rates of drug allergy alerts in inpatient and outpatient settings, and opportunities for improvement. BMJ Qual Saf 2017; 26(3): 217–225.

116.

Nanji

Slight

Seger

, et al. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc 2014; 21(3): 487–491.

117.

Kitchenham

Procedures for performing systematic reviews. Keele: Department of Computer Science, Keele University, 2004.

118.

Boynton

Glanville

McDaid

, et al. Identifying systematic reviews in MEDLINE: developing an objective approach to search strategy design. J Information Sci 1998; 24(3): 137–157.

119.

Higgins

Thomas

Chandler

, et al. (eds) Cochrane handbook for systematic reviews of interventions. Wiley, Glasgow: Wiley, 2019.

120.

Centre for Reviews and Dissemination. Undertaking systematic reviews of research on effectiveness: crd’s guidance for those carrying out or commissioning reviews. CRD Report 4 2nd ed. York: University of York, 2001.

The appropriateness of clinical decision support systems alerts in supporting clinical workflows: A systematic review

Abstract

Keywords

Introduction

Theoretical background

Methodology

Results

Technology

Human

Organization

Process

Outcomes

Discussion

Conclusion

Footnotes

Appendix

Funding

ORCID iD

References