Intensive care unit providers more quickly and accurately assess risk of multiple harms using an engineered safety display

Abstract

Project Emerge took a systems engineering approach to reduce avoidable harm in the intensive care unit. We developed a socio-technology solution to aggregate and display information relevant to preventable patient harm. We compared providers’ efficiency and ability to assess and assimilate data associated with patient-safety practice compliance using the existing electronic health record to Emerge, and evaluated for speed, accuracy, and the number of mouse clicks required. When compared to the standard electronic health record, clinicians were faster (529 ± 210 s vs 1132 ± 344 s), required fewer mouse clicks (42.3 ± 15.3 vs 101.3 ± 33.9), and were more accurate (24.8 ± 2.7 of 28 correct vs 21.2 ± 2.9 of 28 correct) when using Emerge. All results were statistically significant at a p-value < 0.05 using Wilcoxon signed-rank test (n = 18). Emerge has the potential to make clinicians more productive and patients safer by reducing the time and errors when obtaining information to reduce preventable harm.

Introduction

Over the past decade, public and private sectors have made enormous investments in healthcare information technology (IT), much of it focused on adopting electronic health records (EHRs). ¹ Despite improvement of EHR utilization, substantial expected gains in safety and productivity have not been evident. ² A recent estimate described preventable harm as the third leading cause of death in the United States. ³ The EHR appears to have a negligible impact on patient safety and negative impact on clinician productivity and burnout. ⁴ In addition, the omission of evidence-based therapies aimed at mitigating or preventing patient harm has been described as a major cause of preventable harm,^5,6 with patients only receiving about 50 percent of recommended therapies for a given harm. ⁷ Project Emerge seeks to integrate care practice data for preventable harm using systems engineering to improving situation awareness, increasing compliance with best practices, and ultimately improving patient outcome.

Although EHR technology could support safer delivery of care, current features have presented barriers rather than aids to productivity and patient safety. Some of these features are “poor usability, time-consuming data entry, interference with face-to-face patient care, inefficient … work content, and degradation of clinical documentation.” ⁸ A recent analysis found that physicians made more documentation mistakes using an EHR compared to paper charting. ⁹ Kim et al. conducted a systematic review and found that user errors and poor user interfaces blocked receipt of information and caused errors in decision-making. These and other health IT problems were linked to patient harm and death. ¹⁰ Nurses are spending more time with data management tasks and less time with hands-on patient care, ¹¹ which “may be reducing nurse’s attention to safety-critical tasks and creating conditions for errors and unsafe patient care.” ¹² In one study, 86 percent of nurses reported omitting at least 1 of 13 necessary care activities due to lack of time. ¹³

To improve safety, health systems need to implement safety management systems. For example, the use of a checklist for central line insertion improves application of evidence-based therapies and decreases central line–associated bloodstream infections (CLABSI). ¹⁴ However, lack of time is a significant barrier to involvement in safety-related programs.^15,16 Safety programs that are not integrated compete for resources and cannibalize each other rather than synergize interventions to save time and effort and improve their impact. ¹⁷ Recent efforts to eliminate preventable patient harm have modeled health care delivery systems after high reliability organizations (HROs), using systems engineering to purposefully design integrated and safe systems. ¹⁸

An optimal safety system will improve situational awareness in the healthcare environment to address several factors. The system must be accurate in providing the correct information to support decision-making. This contrasts to information cataloged in the medical record which is frequently inaccurate. ⁹ The system must also be reliable in that the output is consistent and can be anticipated by a user over time. The system should be efficient so that users can minimize both the time and the number of interactions with the system (e.g. mouse clicks) to obtain information . Thus, we hypothesized that Emerge would allow providers to more quickly, accurately, and efficiently assess patients’ data for delivery of recommended care. This pilot study answers the question of whether a browser-based health IT system can improve the speed and reliability of these provider assessments.

Materials and methods

Emerge demonstration project

The Emerge system was developed and deployed jointly at the Johns Hopkins University (JHU) and the University of California at San Francisco (UCSF) Medical Center and was funded by the Gordon and Betty Moore Foundation. A transdisciplinary team comprising intensive care unit (ICU) clinicians, systems engineers, computer scientists, human factors engineers, health services researchers, and patient and family advocates began work on Emerge in January 2012. The project was based on the premise that health care needs to evolve from viewing safety as a project seeking to reducing one harm to safety as a management system seeking to eliminate all harms broadly defined, including harms from disrespect. The project goal of Emerge was to improve patient outcome by creating an engineered improvement platform to eliminating seven commonly occurring types of harm in the ICU. The targeted harms were ventilator-associated events (VAEs), pain–agitation–delirium, venous thromboembolic events (VTE), ICU-acquired weakness, CLABSI, care inconsistent with patient goals, and loss of patient respect and dignity. The Emerge team identified 25 recommended, evidence-based care practices known to reduce the risk for these seven harms. We used a systems engineering approach to develop a Concept of Operations (ConOps) that defined the general approach to eliminate avoidable harm. Details of the ConOps have been published. ¹⁹ In brief, for each harm, we defined data elements for the following: what therapies might reduce risk, whether and when patients received those therapies, and what did we learn.

In clinical practice, the processes and tasks needed to prevent avoidable harms are documented in the EHR. To create a comprehensive visualization system for all seven harms, it was necessary to extract and transform the relevant data elements from the Sunrise All Scripts or Epic medical records that were in use at Johns Hopkins Hospital (JHH) and UCSF, respectively. Guided by the ConOps, for each harm, we identified the form and location of specific data elements in the EHR that corresponded to the delivery of recommended therapies to patients at risk for the harm. In many cases, data elements could be in several locations within the EHR with different values or formats. Thus, we created hierarchy tables based on our clinical practice for selection of data for extraction. These extracted data were transformed accordingly and then loaded into the Emerge application and used to populate the real-time visualization display. Data extraction from the medical records occurred hourly at JHH. This limitation of the frequency of data extraction was mandated by the IT services at JHH because a higher frequency of querying the production EHR had the potential to slow the response of the EHR for normal clinical use.

Emerge harm visualization system

A cornerstone of the Emerge ConOps is the identification of risk of harm and compliance with evidence-based therapies. To improve the awareness of both risk and harm, we created an interactive visualization application.

The Emerge visualization application is an interactive user interface that can run on a network-enabled device via a web browser. The design goals were to create a simple and intuitive information delivery system. The design process was led by human factors and software engineers with extensive collaboration with clinicians and patient and family advocates.

The care team portal (CTP) uses a three-tier cascading hierarchy for information display. Tier one provides unit-level information and displays the current ICU census, listing each patient with an overview of their harm risk status. From the census view, a user can select an individual patient dashboard (Figure 1). Tier 2 provides patient-level information with their age, gender, unit location, and the circular harms monitor (Figure 2). The harms monitor uses color coding to indicate which harms the patient is at risk of experiencing and the level of compliance with the best care practices for each harm. Green indicates the patient’s risk for harm was assessed and recommended best practices implemented. Red indicates that the patient did not receive one or more of the recommended interventions. Yellow indicates that more information or processes of care are needed to determine if a true risk of harm exists (e.g. a spontaneous awakening trial has been ordered for the patient, but not yet performed). Gray indicates that the patient is not at risk for this particular harm. Human factors engineers performed extensive testing to develop the presentation graphics including ensuring that the red–green schema could be differentiated by color blind users.

Figure 1.

Emerge census view. This is an example of the top-level census view in the Emerge application which shows patient identifiers, location, and photograph. The red “X” indicates that one or more best practices for eliminating preventable harm have not been completed.

Figure 2.

Emerge harms monitor. This is an example of the harms monitor in the patient-level view of the Emerge application, which uses color coding to show compliance with best practices for eliminating preventable harm. Red indicated one or more best practices were not done, yellow indicated more information was needed to assess risk of harm, and green indicated harm was assessed and recommended best practices implemented.

From the patient-level information (tier 2), a user can select harm-level display (tier 3) to view its detailed information. When a harm type is selected, a condition-specific display (CSD) shows the evidence-based interventions in a format that best represents the delivery of care. For example, time-based therapies are displayed on a 24-h clock and the compliance status reported in hourly blocks of green (compliant) or red (noncompliant). Figure 3 shows the CSD for the eight interventions to prevent VAEs. Six interventions require serial interventions or continuous monitoring and are displayed as sectors in the 24-h clock, while the remaining one-time interventions are presented in the center. When an intervention is selected, in this example, oral care, detailed information, and meta-data, such as values, information source, and time of last update, appear on the screen.

Figure 3.

Condition-specific display for ventilator-associated events (VAE). VAE is one of the seven condition-specific displays. The display uses a 24-h clock to show compliance with best practices for preventing VAE over time. Each ring represents one best care practice therapy associated with preventing a VAE. This figure also illustrates how selection of a best care practice ring (in this case, oral care) opens an information panel to show data noting the status of this therapy

Data platform

Data from the EHR and other connected devices are extracted and transformed through a computing infrastructure to provide integrated health care information in the Emerge application. The data are transferred from the EHR to the Emerge visualization via a novel, open-source platform called Topaz (Figure 4). The frequency and timeliness of data extraction varied for each element and depended on the characteristics of the particular element and its relation to care processes. Some elements were abstracted from the EHR as infrequently as hourly while sensor data were updated every 5 min. The Topaz platform was built to encourage the development of applications for health care improvement through a collaboration between dataFascia Co. (San Francisco, CA) and the Emerge team at Johns Hopkins. The Topaz platform is based on the Apache Accumulo^TM (Apache^® Software Foundation; https://accumulo.apache.org), which is a sorted and distributed data storage technology that has the capability to access cells and modify key–value pairs during data management. Apache Accumulo is based on Google’s BigTable data storage system and leverages Apache’s Hadoop^® and ZooKeeper^TM technologies to provide massively scalable data storage capabilities and incorporate field-level security of protected patient data. ²⁰ Topaz combined the security and scalability of Apache Accumulo with an open application program interface (API) designed specifically for health care data.

Figure 4.

Schematic diagram of Topaz system. The information structure employed by Topaz to ingest and transform the data used to feed the Emerge displays.

Evaluation

We conducted a randomized pilot study to evaluate the usability and efficiency of the Emerge CTP for clinicians. The evaluations were individually conducted in a private conference room in the surgical ICU at the JHH and accessed the Emerge CPT on standalone Apple iPad version 2.0 retina display tablets. We compared the Sunrise enterprise EHR (Allscripts, Richmond, BC, Canada) that was the clinical records system at JHH at the time of the study to the Emerge application. The Sunrise EHR presents data in a flow sheet format, which is comparable to most EHRs. This study was approved by the JHU’s institutional review board.

Clinicians (ICU attendings, fellows, residents, and nurses) from JHH were randomly assigned to follow a test protocol (described below) using either the Emerge CTP or the existing EHR first. Following completion of the protocol, each clinician immediately performed the test in the other system. Prior to randomization, clinicians were shown a 3-min video which demonstrated the information hierarchy, navigation, and color schemes used within the Emerge application. They received no additional training in using their existing EHR. We randomized the initial test system to limit the bias from knowledge gained through their initial exposure. The testing protocol consisted of clinicians using either Emerge or Sunrise to complete a series of data acquisition and interpretation tasks for 28 best practices for five harms: VTE, VAE, pain–agitation–delirium, ICU-acquired weakness, and CLABSI. A stored dataset from an historical ICU patient was de-identified and used for all the testing. Clinicians completed the same tasks using the same patient dataset for both systems. The testing protocol required clinicians to find data elements using either Emerge or Sunrise to answer questions associated with compliance with best care practices. For example, in assessing risk of VAE, participants were asked, “Has this patient had a sedation interruption trial today?” Participants completed a Qualtrics v06/2016 (Provo, UT) survey to assess both the accuracy of the data and the time to complete each task. The survey tool (Supplementary Table 1) presented one data acquisition task to the clinician at a time. Timing began when the task was revealed and ended when a response was provided in the survey tool. Each data acquisition task had only one predetermined correct answer. Screen capture video was recorded using Quicktime v10.4 (Apple, Cupertino, CA) and reviewed retrospectively to track the number of mouse clicks required by clinicians to navigate to the desired data elements.

Measures and analysis

Clinicians’ years of experience with each system are reported as means, range, and standard deviations (SDs). The primary measures were access time (seconds), accuracy (number of correctly identified data elements of 28 total), and the number of required mouse clicks. Paired groups were analyzed using Wilcoxon signed-rank test because the data were not normally distributed. Access times and number of mouse clicks required to access data for each harm, and total time to access the data points for all harms were compared between Emerge and Sunrise EHR. Accuracy data were inadequately powered to assess individual harms, so only total accuracy was analyzed. An alpha less than 0.05 was considered to be statistically significant. Stata v14.1 (StataCorp LP, College Station, TX) was used for all analyses.

Results

There were a total of 18 clinical participants; 4 ICU attendings, 2 ICU fellows, 3 residents, and 9 nurses. Of these, 10 clinicians were randomized to use the Sunrise EHR first and 8 used the Emerge CTP first. Mean experience among clinicians was 6.2 years (min = 1, max = 12, SD = 3.3 years) with their EHR and 10.9 months (min = 0, max = 24, SD = 9.8 months) with the Emerge CTP. There was no significant difference in the outcomes between the groups randomized to begin with the EHR versus the Emerge CTP (data not shown).

The median time to access all data elements with the Emerge CTP was 481 s (interquartile range (IQR): 357–599 s). This was significantly faster than the EHR median time of 1098 s (IQR: 806–1360 s; Z = 3.68, p = 0.0002) (Figure 5).

Figure 5.

Comparison of time needed to access data between Emerge and EHR. Box-plot diagram showing the time required to access data in the EHR versus Emerge, for each harm and an aggregate total of all harms. Boxes represent the middle interquartile ranges with divider bar at the median. The outer quartiles are represented by the whiskers. n = 18 for each box-plot.

The aggregate median number of mouse clicks needed to access relevant information in Emerge CTP was 38 (IQR: 33–56) which was significantly less than the EHR median of 96 (IQR: 78–115, Z = 3.73, p = 0.0002) (Figure 6).

Figure 6.

Comparion of usability as measured by mouse clicks between Emerge and EHR. Box-plot diagram showing the number of mouse clicks required to access data in the EHR versus Emerge, for each harm and an aggregate total of all harms. Boxes represent the middle interquartile ranges with divider bar at the median. The outer quartiles are represented by the whiskers. n = 18 for each box-plot.

The testing protocol required clinicians to find 28 data elements. Clinicians more accurately identified that these data in Emerge (median correct 25; IQR: 23–27) were significantly better than in the EHR (median correct 22; IQR: 20–23; Z = 3.60, p < 0.0003) (Figure 7).

Figure 7.

Comparison of accuracy of information acquisition between Emerge and EHR. Box-plot diagram showing the number of data points correctly accessed in the EHR versus Emerge. Boxes represent the middle interquartile ranges with divider bar at the median. The outer quartiles are represented by the whiskers. n = 18 for each box-plot. There were 28 total data points. The comparison had a statistically significant differnce at p < 0.05.

Discussion

The goal of the Emerge system was to improve the awareness of clinicians to the patients’ risk of multiple harms so that they will apply best care practices to reduce or eliminate harms. ²¹ Because users often cite data presentation and “number of clicks” as reasons for their failure to consolidate information, ²² we developed Emerge with the plan of presenting harms data in a concise, intuitive, graphical format.

The clinicians we tested had an average of about 5 years of experience using the existing EHR and were asked to perform data acquisition that they routinely perform as part of their daily ICU duties. As such, one would expect these highly trained, capable clinicians would be expertly facile in this function. Instead, clinicians spent less time maneuvering through screens and accessing data with the novel Emerge CTP as compared to their EHR system. Similarly, clinicians were asked to perform these tasks in a controlled environment with the knowledge that the results would be scored. Thus, their performance using the EHR should reflect a best possible, or ceiling, accuracy. We predict that the real-world accuracy of data abstraction would only be inferior to what was measured in this study. Despite this, clinicians’ accuracy in identifying the correct information was significantly higher with Emerge compared to the EHR. This suggests that extensive experience using a clinical records system will not overcome a design that impairs specific data consolidation tasks.

It is recognized that an understanding of best care harms reduction practices does not necessarily translate into improved clinical practice. For example, a low tidal volume ventilation strategy was demonstrated to improve outcomes in patients with ventilator-associated lung injury in 2002. ²³ Yet, despite this knowledge, adoption of this therapy to specific patient populations was only 41 percent when surveyed in 2012. ²⁴ One method for improving compliance with best care practices is the adoption of safety checklists. The coupling of checklists with clinical practices has had remarkable and enduring success in certain areas such as the reduction in CLABSI.^25,26 However, as we have increased the number of harms to which we can apply bundles of best practices, our adherence has degraded, a phenomenon known as “checklist fatigue.” ²⁷ Improved information systems designed for enhancing patient safety may offer new ways to reduce the burden of monitoring and ensuring safe patient care. For example, Starmer and Giuse ⁶ developed an electronic ventilator-specific dashboard to improve adherence to best care practices for care of ventilated patients. In the Emerge project, we intentionally developed a system that simultaneously addressed multiple harms so that one particular patient harm would not compete for the attention of the clinical team.

Providers in the ICU face substantial time pressures, which can directly and negatively impact patient safety. ²⁸ The Emerge system saved a mean of 10 min per patient, while also improving the quality of safety assessments. If we extrapolate this 10 min to a typical 14-bed ICU, we project that a clinician who performs comprehensive safety assessments for all patients on the unit could save at least 2 h each day.

The reliable delivery of recommended therapies, whether from errors of omission ⁵ or misinformation, ¹⁰ is a prominent cause of preventable harm. A solution to reliably deliver these therapies is not easy and current EHRs do not adequately support clinicians in monitoring compliance. EHRs require navigation through multiple flow sheets and mental computation of medical heuristics and algorithms to determine compliance with care bundles. The Emerge system collects and integrates data from the EHR and other sources and converts it into information that is easily interpreted on a simple display. Given the lack of visual displays and poorly communicated information in EHRs, it is not surprising that such systems have had little impact on quality or productivity. ²⁹ While electronic data are required for all information systems, the systems themselves must add value by transforming this data into information that meets the needs of the end user, in this case the clinician. A systems engineering mindset would reconceive of the EHR as being a necessary component of a greater information ecosystem.

During the development of Emerge, we encountered a number of challenges that would be generalizable to the implementation of data information system aimed at reducing preventable harm. When building Emerge, we found it very challenging to extract valid electronic data from the EHR to monitor compliance. Data are embedded in EHRs for various documentation purposes and are often distributed across multiple tables and data fields so that defining the proper data elements to drive Emerge and evaluate compliance requires detailed analysis by the engineering team. After identifying how and where the data are stored, the validity of the data must be considered. This can be complex as informatics scientists and quality researchers define validity differently, often resulting in confusion and sometimes conflict. Informaticists view validity by the accuracy of their code—did they pull the data the user requested? Researchers view validity by whether the measure of compliance correlates with the actual therapies that the patient received. ³⁰ An engineered health system could define data elements that meet both these requirements. Reliable processing of this data to drive patient-safety efforts requires a platform such as Topaz, which can extract and transform data from a variety of sources and perform real-time analytics.

There are many potential harms that are not currently addressed within Emerge. These harms must also be assessed and tracked using time-consuming and labor-intensive methods. Indeed, an average hospital patient with multiple chronic diseases may be at risk for over a dozen harms that require scores of therapies which necessitates that clinicians perform hundreds of clicks and remember reams of data. ³¹ This is far beyond our capacities, and as a result, awareness and accountability are often lacking. ³² With the amount of time and expertise required to complete these tasks, it is not surprising that reliably delivering preventive therapies and the eliminating harms are often perceived as impossible tasks.

We recognize several limitations to our study. First, we sampled a small number of clinicians in this study, thus our result may not be generalizable. However, we believe that this sample to be valid because they were more likely to use the EHR well than an average user, because of their experience using the EHR and that they knew that they were being evaluated. Both conditions should have biased the evaluation in favor of the EHR. We have tacitly assumed that improved user performance will result in better provision of best care practices and consequently better patient outcomes; however, this manuscript does not address this specific question. A further limitation of our work is that Emerge has only been deployed in a few ICUs in two academic medical centers and it may not fit the workflow of other settings.

Conclusion

The current EHR systems are a vital component of clinical care. However, they were not designed and built with the explicit goal of eliminating patient harm. Rather, the EHR has evolved to replace traditional medical records documentation. In contrast, Emerge was conceived for the specific purpose of managing patient-safety risks and was designed with significant end-user input. Our data show that Emerge improves information retrieval efficiency and awareness. This should result in secondary improvements. This study suggests that the Emerge system may be an effective complement to the EHR for units that wish to more efficiently and effectively manage patient-safety activities.

Supplemental Material

Supplementary_Table_1 – Supplemental material for Intensive care unit providers more quickly and accurately assess risk of multiple harms using an engineered safety display

Supplemental material, Supplementary_Table_1 for Intensive care unit providers more quickly and accurately assess risk of multiple harms using an engineered safety display by Mark Romig, Howard Carolan, Alan Ravitz, Hildy Schell-Chaple, Edward Yoon, Cindy Dwyer, Kevin Thornton, Michael Gropper, Peter J Pronovost and Adam Sapirstein in Health Informatics Journal