Abstract
In this study, we evaluated the usability aspects of medication-related clinical decision support systems in the outpatient setting. Articles published between 2000 and 2016 in Scopus, PubMed and EMBASE were searched and classified into three usability aspects: Effectiveness, Efficiency and Satisfaction. Using Van Welie et al.’s usability model, we categorized usability aspects in terms of usage indicators and means. Out of the 1999 articles, 24 articles met the selection criteria of which the main focus was on reducing inappropriate medication, prescription rate and prescription errors. Evidence could mainly be found for Effectiveness and showed high rates of positive results in reducing medication errors. To date, the effects of Efficiency and Satisfaction of clinical decision support systems regarding medication prescription remain understudied. Usability aspects such as memorability, learnability, adaptability, shortcuts and consistency require more attention. Studies are needed for better insight into the user model and to design a knowledge/task model for clinical decision support systems regarding medication prescription.
Introduction
Medication errors are a great expense for health-care systems, both clinically and economically, and they occur frequently.1–5 Hospitalizations due to medication errors cause a substantial increase in length of stay and disability at discharge.6–8 These medication errors can occur during any part of the medication process: taking history, ordering, dispensing medicine, administration management or surveillance. 9 However, the ordering step in this process is where errors happen most often. 8 These errors can be attributed to many different reasons, of which failure to monitor patients, 10 lack of appropriate knowledge and high workload 11 are among the major causes.
Computerized physician order entry (CPOE) and e-prescribing (eRx) have been shown to be valuable resources in avoiding these medication errors.7,12,13 An estimation was made that 5 million medication errors in the outpatient setting could be avoided with e-prescribing. 14
Clinical decision support systems (CDSSs), defined as “systems designed to aid in making decisions in health care,” 15 have been proven to have positive effects on clinical practice, with safety and quality benefits being among these effects.16–18 However, the effect of some of the aspects of CDSSs, such as usability, defined by the International Organization for Standardization (ISO) as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use,” 19 has not yet been studied systematically. Effectiveness is defined as the “accuracy and completeness with which users achieve specified goals,” while efficiency is understood as “resources expended in relation to the accuracy and completeness with which users achieve goals.” Satisfaction is the subjective “degree to which user needs are satisfied when a product or system is used in a specified context of use.” The notion of efficiency can encompass any kind of resource, including money or knowledge, but the concept is normally limited to indicate the amount of time spent on a certain task.
It is of importance to study the consensus regarding usability, as it is one of the success criteria of implementing and utilizing CDSSs.20,21 This success depends on the design, functionalities and features, and deals with the needs of the users and the system behavior, which are all incorporated in the software.16–18 Improving usability is crucial, as poor usability is a core barrier for successfully integrating a system. 22 There is proof that usability can both reduce costs and increase profits among others by reducing training, reducing errors and increasing the productivity of users. 23 However, previous studies showed that usability aspects of CDSSs and integration of these systems within e-prescribing systems still need improvement in different settings.24–27 A list of 168 instances of usability flaws of medication-related alerting functions is already generated, which can act as a general check-list of usability of medication-related alerting functions during their design or evaluation process. 27 However, there is no systematic study of the usability effect of medication-related decision support systems in the outpatient setting where usability improvements are needed regarding the effectiveness, efficiency and satisfaction aspects. Since this evidence is also lacking for the inpatient, a separate study is being performed specifically for this setting, the findings of which will be described elsewhere.
In order to provide operationalized measures of usability for evaluation purposes, Van Welie et al. 28 developed a model dividing the three aspects into subtopics (Figure 1). The model consists of three related layers. The first one comprises the three aspects of usability, that is, Effectiveness, Efficiency and Satisfaction. The second layer relates usage indicators to these aspects, such as Learnability, Memorability and Performance Speed. The third layer, finally, provides means by which the usage indicators can be measured. These include consistency, the availability of undo operations and the presence of Feedback.
Keywords used in the search query composed of four separate parts. (A) First part, keywords relating to electronic prescription and decision support. (B) Second part, medication. (C) Third part, the outpatient setting. (D) Fourth part, usability. The four parts were combined using “AND” statements resulting in the final query.
Objective
Following the above-mentioned problems with CDSSs, and their documented usability issues, we believe a contemporary systematic overview of the hitherto performed studies is required for the outpatient settings. The objective of this study is therefore to systematically summarize the results of studies evaluating usability aspects of CDSSs integrated in e-prescribing systems in the outpatient setting. Since such a summary is also lacking for the inpatient setting, a separate study is performed specifically for that setting, the findings of which will be described elsewhere.
We aim to summarize the results within the three aspects of usability and to explore the current evidence of usability based on Van Welie’s usage indicators and means in CDSS evaluation research. These indicators are then analyzed to discover which usability categories in the ISO standard need further investigation.
Method
Approach
The study was designed as a systematic literature review, following a stepwise approach. In Figure 1, the steps in our study design are depicted with regard to the query composition, as well as the various inclusion and exclusion rounds of the study.
In the first step, an initial literature search was conducted through a predetermined query format (A.H.). The query was composed of four separate parts: (A) the first part, having keywords relating to decision support; (B) the second part, medication; (C) the third part, outpatient setting; and (D) the fourth part, usability. The four parts were combined using “AND” statements resulting in the final query.
In the second step, all article titles and abstracts were independently screened by two reviewers (A.H. and B.K.) to select those qualified for inclusion. Both reviewers would look for keywords and techniques which were clearly attributed to the query composition and main subjects of the study. If any discrepancy or confusion arose then consensus among the two reviewers would be found, or if needed a third reviewer was consulted (M.A.). Inter-rater agreement was measured by Cohen’s kappa to measure the acceptance rate of articles between the two reviewers. Cohen’s kappa is a nominal scale agreement which treats all disagreements equally and was therefore judged to be a good way to ensure consensus between reviewers. 29
In the third step, articles were reviewed using a structured format (A.H.): after reading the studies thoroughly, the study design, aspect of usability, type of effect, continent and significance were collected. If articles’ contents did not satisfy our (A.H., B.K., M.A., and M.L.) requirements for inclusion, they were excluded from the study.
The final step consisted of summarizing articles’ contents pertaining to usability (A.H.).
Inclusion and exclusion criteria
Predetermined queries were performed on Scopus, PubMed and EMBASE, on articles published between 1 January 2000 and 31 January 2016. Non-English languages articles, non-medical disciplines, animal-related articles, opinion papers, letters, study protocols and reviews were excluded. The final search queries were executed on 4 February 2016.
Articles were selected for inclusion if they evaluated aspects of usability of CDSSs related to medication in an outpatient setting. Reasons for exclusion included studies being patient-centered instead of clinician-centered, having been performed in another setting than outpatient, studying cost-effectiveness or studying e-prescribing system without CDSS.
Analysis
For each included paper, details were collected about when and where the study was conducted. Information about the study approach, such as the number of participants and the method, was collected when available. Using this information together with the hierarchy of study designs developed by the University of California, San Francisco—Stanford Evidence-Based Center, 17 each study was further categorized into one of four categories described in Table 1.
Categories of study designs according to hierarchy of study designs developed by the University of California, San Francisco—Stanford Evidence-Based Center.
In order to answer our question, we synthesized our results per usability category: Effectiveness, Efficiency and Satisfaction, based on ISO definition. We studied the results based on their outcome positive, negative or none effect).
Positive results were reported when the outcome measure resulted in an enhancement of an aspect of usability. Negative results were reported when the outcome measure resulted in an impairment of an aspect of usability. Mixed results were reported when the results reported both positive and negative outcomes for a particular usability aspect.
In order to explore the relations between the three aspects of usability and Van Welie’s usage indicators and means, we synthesized each study following his model in each category (Figure 2).
Van Welie’s layered model of usability. 28
Results
General results
After running the final query, a combined result of 1999 articles was acquired. Screening the titles and abstracts of these articles led to a total of 96 articles for full text review. Of which, 24 (25.0%) articles were included in the final selection. The calculated Cohen’s kappa resulted in a score of 0.72 which is fair to good.
Out of the 24 articles, 11 (45.8%) were observational studies without controls (OSWoCs), 7 (29.2%) were randomized controlled trials (RCTs), 3 (12.5%) were non-RCTs and 3 (12.5%) were observational studies with controls (OSWCs). Figure 3 illustrates the distribution of the study designs over the years, showing that no clear trend is present. The highest number of the same study design in a given year is two, which illustrates a lack of trend throughout the years. In total, 17 studies (70.8%) were performed in the United States, 5 (20.8%) were performed in Europe and 2 (8.3%) in Canada. Of the 24 articles, 21 were performed after 2005 (87.5%) and 12 of these were performed after 2010 (50.0%).
Distribution of study designs per year.
Table 2 summarizes the results based on study design and the type of effect within the main usability aspects. In this table, a total of 31 combinations could be found in the articles, which is possible because of different usability aspects being studied in part of the articles. A total of 19 out of the 31 combinations (61.3%) could be categorized in category Effectiveness, 6 (19.4%) in Efficiency and 6 (19.4%) in Satisfaction. Of which, 22 (71.0%) studies found a positive effect, 1 (3.2%) study found no effect, 6 (19.4%) studies found a negative effect and 2 (6.5%) studies found mixed effects (meaning both positive and negative results were found). In the next section, the results for each part of usability will be described. Table 3 summarizes the study characteristics and extracted information from all the included studies regarding usability aspects, outcome measures, usage indicators and means.
Effects of each study compared within the categories.
OSWoC: observational study without control; OSWC: observational study with control; RCT: randomized controlled trial.
Study characteristics and information including the usability aspects, outcome measures, usage indicators and means.
(a)—Effectiveness, (b)—Efficiency and (c)—Satisfaction.
Statistical significance.
Results per category
Effectiveness
Overall, 19 studies (79.2%)20,30–47 studied Effectiveness; 14 of these studies (73.7%) were performed in the United States, 3 in Europe (15.8%) and 2 in Canada (10.5%). Out of the 19 studies, 6 studies were RCT (31.6%), 3 were non-RCT (15.8%), 3 were OSWC (15.8) and 7 were OSWoC (36.8%).
From the 19 studies in this category, 13 had mainly focused on reducing inappropriate medication, which encompasses unsafe prescriptions;35,36,38,39,46 prescription rate, which encompasses the decrease in undesirable medicine or increase in desired medicine;32,33,40,41 and prescription errors, which encompasses true errors made during prescription.31,43–45 Of the 5 articles (26.3%) that have focus on reducing inappropriate medication,35,36,38,39,46 4 found significant positive results which showed a decrease in prescription of unsafe prescriptions35,36,38,39 and 1 found no effect regarding Effectiveness. 46 The 4 articles (21.0%) that had focus on the prescription rate32,33,40,41 resulted in positive effects by increasing the prescription rate for desired medication33,41or decreasing for undesired medication 40 and 1 found a significant decrease in interacting prescription rate of undesired medication. 32 And in the subcategory prescription errors, all 4 articles (21.0%) resulted in significant decline of prescription errors.31,43–45
From the 19 articles, the outcome accuracy increase was measured, which was found when a new system could increase the accuracy of useful information used by having the sensitivity of the e-prescribing increase, specifically within 3 articles (15.8%).20,34,37 1 found a significant positive effect 37 and the other 2 found a positive effect; however, the articles did not address significance.20,34 Finally, 3 articles measured different outcomes.30,42,47 1 measured an increase in treatment goals set by doctors 42 and one an increase in appropriate decisions made by doctors. 30 They found a positive effect on Effectiveness.30,42 One found a negative effect with most of the participants agreeing that the system did not contain all necessary functionalities yet (completeness). 47
When we analyzed the results based on Van Welie’s usage indicators, we found 14 (73.7%) studies30–33,35,36,38–41,43–46 focused on Safety and Error of which 13 found a significant positive effect. Memorability is never measured in the selected studies. 5 studies (Safety and Error) focused on accuracy increase as well as increase in treatment goal and completeness of the system.20,34,37,42,47 We did not categorize them under the Effectiveness indicator Safety and Error.
Of the previously mentioned 14 studies, 4 fell in the Feedback means of which all found significant positive results,30,33,39,41 9 fell in the Warnings means of which 8 found significant positive results31,32,35,36,40,43–46 and 1 fell in both the Warnings and Feedback means which found significant positive results. 38
Efficiency
The 6 articles (25.0%) categorized under Efficiency studied mainly the amount of time spent on using the system and the mental burden of the clinician.30,34,47–49,51 4 of the 6 studies were performed in the United States (66.7%) and 2 in Europe (33.3%). 5 studies could be categorized as an OSWoC design30,34,47,49,51 and 1 study could be categorized as RCT. 48 2 articles (33.3%) are focused on the mental burden34,51 but they differ in outcome. 1 study found a positive effect by decreasing mental burden by changing alert criteria 34 and the other study found a negative effect because the amount of alerts increased the mental burden of participants. 51 The 4 articles (66.6%) focusing on time30,47–49 could be divided into 2 articles that found a negative effect because the implementation of the system caused the doctors to spend more time entering data30,49 of which 1 was significant, 49 1 article that found a significant positive effect because using the new system decreased time spent 48 and 1 article that found results that were both positive and negative because participants mentioned the system decreased time spent recording data, but still took more time because high quantities of data still needed to be input in the system. 47
Further analysis according to Van Welie et al.’s 28 usage indicators for Efficiency, which are Performance Speed and Learnability, was performed. Performance Speed was inherent in the outcome measures: mental burden and time spent;30,34,47–49,51 Learnability, however, was only partly studied in 2 articles48,49 and found that the system was difficult to master and did not improve the workflow and efficiency in 1 article, 49 while the other article mentioned some progress regarding Learnability 48 but did not show concrete results.
Finally, by analyzing the means layer of Van Welie et al.’s 28 model (Figure 2), a distribution can be found within the means of all studies. Only 2 studies have the same means in Efficiency. We found 1 study having the Shortcuts as means, 51 1 having both the Consistency and the Shortcuts means, 34 1 having the Adaptability means, 30 1 having the Adaptability and Shortcuts means, 47 and finally, the 2 studies having the Consistency, Task Conformance and Adaptability means.48,49 This distribution shows a big variation in studies performed on the Efficiency aspect. To summarize, 3 studies had a focus on the Shortcut means, 2 on the Consistency means, 3 on the Adaptability means and 1 on the Task Conformance means.
Satisfaction
6 articles were categorized under Satisfaction (25.0%). 3 out of 6 were performed in the United States (50.0%) and 3 were performed in Europe (50.0%). 5 articles (83.3%) included in the Satisfaction category were observational studies without controls21,30,47,49,50 and 1 (16.7%) was an observational study with control. 20 Our results showed that 4 articles were quantitative studies (66.7%) focusing on measurable results obtained from questionnaires20,21,30,47 and 2 articles were qualitative studies receiving results from interviews (33.3%).49,50
Out of the 6 articles in this category, three used the system usability scale (SUS) 21 with mixed results.20,21,30 1 study resulted in a negative effect on Satisfaction regarding the system, with a below-average SUS score of 63.25 30 (68 being the average score of SUS). 52 The other studies focused on the redesign of the graphical user interface (GUI) of a CDSS and had an increase in the SUS score.20,21 1 study also used the Center for Health Care Evaluation Adapted Provider Satisfaction Questionnaire (CHCE-PSQ) where it increased the score on a 5-point scale from 4.0 to 4.6. 21 1 article has positive results regarding Satisfaction which was found by 79 percent of participants agreeing with the statement “I feel comfortable using the system”; however, only 54 percent agreed with the statement “Overall, I am satisfied with how easy it is to use the system”; therefore, the positive effect seems to be average. 47
2 articles were more focused on the advantages and disadvantages of the CDSS;49,50 both studies have mixed results, but 1 is mostly negative and focused mainly on the disadvantages, 49 while the other did not lean toward positive or negative and covered advantages and disadvantages equally. 50
The studies found in the Satisfaction usage indicator are the only studies in the usability aspects to have the same means, Adaptability and Undo.20,21,47,49,50 However, the effects found in the studies in the Satisfaction aspect were very divergent.
Discussion
Main findings and interpretation
In this review, we studied 24 articles that included at least one aspect of the usability of medication-related CDSSs in the outpatient setting. Our findings showed positive results for Effectiveness in 90 percent of the studies. Overall, 14 studies focused on Safety and Error explicitly. Of these studies, the three main outcome measures were reducing inappropriate medication, improving prescription rates and reducing prescription errors. The high percentage of positive results for Effectiveness differs from Efficiency and Satisfaction. Within Efficiency, 33.3 percent of studies had positive results. Within the category Satisfaction, 50 percent of the studies resulted in a positive effect, of which 33.3 percent was found in SUS-related outcome measures and 17 percent in satisfaction rating outcome measures (this however was not a strong positive effect). Our results also showed that most of the studies were performed in the United States and slightly more than the half were observational studies, which is the lowest level in the study designs according to the hierarchy presented in Table 1. Satisfaction does depend on subjective outcomes according to the ISO definition, which makes it difficult to design a study that gives higher level of evidence. Nevertheless, a stronger study design is recommended at least for studies focusing on Effectiveness and Efficiency to increase the validity of the usability studies. Only 39 percent of the studies focused on Efficiency and Satisfaction and showed very heterogeneous results. Therefore, more studies focused toward these aspects of usability are needed.
Our results showed that some of the usage indicators of Van Welie et al.’s 28 model (Figure 2), such as Memorability and Learnability, were barely or not at all studied. Memorability and Learnability are essential parts of Effectiveness and Efficiency and therefore should be included and elaborated upon in future research regarding usability.
We particularly studied the means layer of Van Welie. We found that the means differ greatly between studies, and that most studies focused on the Warnings rather than the Feedback means. These means showed 88.9 and 100 percent significant positive results, respectively. A possible reason might be giving feedback is a task which needs more knowledge of the situation and therefore generating warnings is easier to implement. We hypothesize that generating feedback can lead to improving the CDSSs, because this might lead to a direct influencing and supporting the decision-making process of the physicians, instead of only warning them when they do something wrong. We only found one study that focused on the combination of both Warnings and Feedback, which found a significant positive effect. More studies are needed to investigate the differences between the effects of warnings and feedback forms. In the category Efficiency, 4 studies measured time spent and two measured mental burden, both inherent to Performance Speed. The mental burden measurement found 50 percent positive results and the time spent measurement found 50 percent negative results.
The studies in the Satisfaction category all focused on the same means: Adaptability and Undo. However, the effects found in the studies, from the Satisfaction perspective, are very divergent. The main reasons for this could be that all the systems in these studies were slightly different in use and the authors used different types of questionnaires/tools to measure the user satisfaction.
Comparison with other studies
Four comparable studies were found that evaluated the usability of CDSSs in an outpatient setting.15,27,53,54 These studies differ from ours in some ways, varying from only studying one aspect of usability 54 to having studied CDSSs which did not only focus on medication. 53 Marcilly et al. 27 analyzed usability flaws instead of usability as a whole. Eslami et al. 15 focused on medication-related CDSSs in combination with CPOE systems. However, the study focused more on the CPOE systems than on CDSSs and did not specifically focus on usability, but only addressed it briefly. Our study uses the entire definition of usability as defined by ISO 19 and focused in depth on sub-aspects of the usability components. Despite the differences, Eslami et al. 15 found that the study could not yet provide enough evidence and that CPOE could provide enough benefits related to usability. Bright et al. 53 performed a thorough research and covered efficiency and satisfaction. However, the studies in Bright et al. 53 were selected if the CDSS aided in decision-making at the point of care or specific care situations and were not specifically medication related or outpatient related. They found positive effects regarding CDSSs and improving health-care process measures, but had difficulties with finding useful results in efficiency outcomes, among others.
The study by Bryan and Boren 54 only focused on effectiveness of CDSSs in primary care and found a positive effect for CDSSs that focused more on improving quality of care. They found positive results regarding effectiveness. However, our study focused on a more elaborate definition of usability (from the Van Welie layers of usability) and only included studies specifically focusing on medication.
The study by Marcilly et al. 27 focused on usability flaws, which were defined as the violations of usability design principles and are additionally known as usability heuristics or usability criteria. In addition, they focused on alerting systems that supported the prescribing of medications, which were used in general hospitals or in primary care general practices. Aligned with their aim, they generated a list of flaws that were found in the literature, which could serve as a check-list for checking the usability of medication-related alerting functions during the design or evaluation process of such systems. In our study, we focused on the outpatient setting and studied the means that measure the usage indicators, which then lead to generating knowledge about the level of usability (Effectiveness, Efficiency and Satisfaction). We also investigated which measures and categories need to be given more attention in future studies, regarding the usability pillars of the ISO definition.
Limitations and strengths
To the best of our knowledge, this is the first study that performed a systematic review solely focused on CDSSs and the usability aspects related to medication prescription, specifically in the outpatient setting. Multiple reasons that limited our search and results were as follows: first, although we performed an extensive search, it is still plausible that we missed articles. In addition, we chose to include English-language studies only, which might affect the results with regard to countries of origin. Second, we used the ISO definition to categorize our findings and used the Model of Usability introduced by Van Welie et al. 28 (Figure 1) as one of the two important known models; however, there are other usability models that could be used and could result in studying different aspects of usability. Finally, the included studies were too heterogeneous to perform a meta-analysis, mainly because of the variety in research designs (more OSWoC than RCT studies); execution of experiments and measuring the outcomes was rarely done in the same way. To further improve upon the quality of research within the usability domain, more research incorporating the RCT study design would be a valuable addition. However, it should be noted that to perform usability research of high quality, implementation of multiple types of software is necessary (dummy software for control groups and software to be implemented) which requires more resources.
Specifically, studying effectiveness was beyond the scope of this review. We analyzed the studies based on the third layer of Van Welie: “means”. It should be mentioned that these means were not the only possible means to achieve improvement in the usage indicators; according to Van Welie et al., these were a given set of appropriate means at the moment of the study, but other means might also be possible.
Conclusion
Evidence for usability of medication-related CDSS could be mainly found for Effectiveness and showed a high percentage rate of positive results in reducing medication errors, which is in line with previous studies. The results in the categories Efficiency and Satisfaction, however, were too divergent to draw a conclusion and these two categories remain understudied. Memorability and Learnability were hardly studied, together with Consistency, Task Conformance and Shortcuts. Studies focusing more on these effects and of higher study design level are needed for further investigation. Warnings were the most found mean. To confirm the usefulness of Warning and Feedback, further investigation is needed. Future studies are needed to generate better insight into the user model as well as design knowledge and task model for CDSS regarding medication prescription in the outpatient setting.
Footnotes
Acknowledgements
M.A., M.L. and A.H. conceived the study and study design; A.H. and B.K. carried out the search and data analysis under supervision of M.A. and M.L.; M.A. coordinated the study; A.H. drafted the manuscript; and A.H., B.K., M.A., M.L. and M.M. participated in the interpretation of data and critically read and revised the manuscript.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship and/or publication of this article.
