Development of a digital clinical pathway for emergency medicine: Lessons from usability testing and implementation failure

Abstract

Clinical pathways are used to support the management of patients in emergency departments. An existing document-based clinical pathway was used as the foundation on which to design and build a digital clinical pathway for acute chest pain, with the aim of improving clinical calculations, clinician decision-making, documentation, and data collection. Established principles of decision support system design were used to build an application within the existing electronic health record, before testing with a multidisciplinary team of doctors using a think-aloud protocol. Technical authoring was successful, however, usability testing revealed that the user experience and the flexibility of workflow within the application were critical barriers to implementation. Emergency medicine and acute care decision support systems face particular challenges to existing models of linear workflow that should be deliberately addressed in digital pathway design. We make key recommendations regarding digital pathway design in emergency medicine.

Introduction

New health informatics systems have the potential to improve the efficiency, effectiveness, and safety of healthcare delivery. The emergency and acute medical care environment presents a particular challenge to such systems because the clinical problems encountered are often of great potential harm to the patient, can be time critical, and frequently involve difficult decision-making and multidisciplinary collaboration. Emergency medicine has been described as a complex system with unpredictable clinical conditions and work patterns, a vast decision space, and incomplete evidence. ¹ Task analysis has shown that in addition to episodes of clinical and non-clinical work, emergency medical care is characterized by frequent interruptions and multi-tasking. ²

One common solution to this complex environment has been to create clinical guidelines, which are expert syntheses of evidence-based medicine and define a standard for patient care. When adapted for local use, these become clinical pathways which serve as a pragmatic guide for the clinical user and a map of clinical workflow that describes the actual patient journey through a local health system. Clinical pathways aim to standardize care so that it consistently equates to best practice, resulting in improved patient safety and resource allocation and a reduced need for re-negotiation between services. Such is the success of the pathways approach that one suite of web-accessible clinical pathways developed in Canterbury, New Zealand is now used for a catchment of over 23 health jurisdictions including at least 15 million people around the world. ³ Importantly, there has been a high level of end-user acceptance and uptake of such pathways. Local audit data from a single acute hospital in Canterbury show approximately 45,000 “hits” online per month on pathways by clinicians

Computerization of clinical guidelines and pathways has led to a number of different approaches to designing digital clinical pathways from electronic versions of paper pathways ⁴ to more sophisticated computer-interpretable guidelines and clinical decision support systems (CDSSs). ⁵ Digital clinical pathways are becoming more common in clinical practice with a growing evidence base to support their adoption in many clinical domains. Computerized clinical guidelines have been shown to improve processes of care, particularly, when presented as part of clinician workflow, ⁶ and CDSSs have demonstrated benefits for a variety of outcomes.^7,8 A number of CDSS methodologies, ⁵ design principles, ⁹ and sociotechnical dimensions ¹⁰ are outlined in the literature.

One example of a clinical pathway in common use is the rapid cardiac chest pain pathway, for the structured assessment of chest pain, the most common presenting symptom of myocardial infarction. Chest pain is a problem that faces multitudes of patients per year, over 6 million in the United States alone. ¹¹ Patients with symptoms such as chest pain that are suggestive of myocardial infarction compose approximately 5 percent to 10 percent of annual presentations to emergency departments (EDs) and up to 25 percent of hospital admissions. Correct identification, management, and disposition of these patients are paramount as a missed diagnosis of myocardial infarction that can lead to death or other adverse outcomes. ¹² Hence, both the frequency and clinical significance of the problem make a chest pain pathway very compelling to the emergency physician. The chest pain pathway in use in Christchurch Hospital, New Zealand has been prospectively validated in a randomized controlled trial as an accurate means of safely identifying those at risk while leading to shorter hospital stays. ¹³ It includes a score: the Emergency Department Assessment of Chest Pain Score (EDACS) which allows risk prediction based on demographics, patient history, and the nature of chest discomfort. ¹⁴

In this project, the chest pain pathway in use in the Emergency Department of Christchurch Hospital (a metropolitan regional, general, and tertiary hospital) was selected as a pathway suitable for digital translation. It was also decided that there was a need for this because of several inefficiencies identified with current use. These included (1) parallel use of manual and digital record-keeping, (2) frequent duplication of key tasks, (3) repeated patient information requests, (4) manual calculation and entry of key risk scores, and (5) failure to collect key data for audit and research.

The aim of this project was to combine clinical workflow, decision support, documentation, and research within the electronic health record (EHR) by creating a digital clinical pathway for acute chest pain. User testing of the working prototype would dictate whether the pathway was suitable for deployment into the live clinical environment.

Methods

Setting

Christchurch Hospital is the general and tertiary hospital for approximately 450,000 people. The ED has approximately 100,000 patient attendances per year. Patients who present to the ED with possible cardiac related chest pain are usually entered into a cardiac chest pain clinical pathway containing specific steps and guidance.

Design process

We identified 10 separate paper or electronic conceptual and cognitive processes that could be combined by creating a collaborative digital clinical pathway for acute chest pain. These included an initial emergency medicine assessment, chest pain pathway checklist, risk score calculation, blood test timing calculation, cardiology specialist team review, nursing handover, cardiac stress test referral, final clinical review, encounter summary, and research record. These processes occur across dimensions of location, team, task, documentation, and research data collection. A schematic of existing processes is shown in Figure 1.

Figure 1.

Workflow for the assessment of chest pain depicted in terms of the patient journey across time, service, location, tasks, notes, and research. All can be catered for with a single clinical pathway based tool.

Design of the digital pathway was iterative and clinician-led with frequent consultation with emergency, cardiology, and technician teams. Personal experience of the authors and ethnographic observation of clinicians at work in the ED were used to optimize a decision sequence. The governance for the clinical content of this chest pain pathway project resided with the team of clinical experts from emergency medicine and cardiology groups.

Evaluation

Following pilot pathway design and technical validation, user testing was performed in a test environment that emulated real clinical workstations. Emergency medicine and cardiology doctors were given detailed scenarios based around simulated patients, designed to elicit different cardiac risk outcomes (Figure 2). A “think-aloud” method ¹⁵ of recording responses during testing was used. Participants were invited to interact with the system while they entered their simulated patient data and followed the digital pathway. Subjects were asked to ““think aloud” or verbalize their thoughts as they interacted with the system as they carried out the task. These verbalizations were recorded, and subsequently, thematic analysis was performed by the author and design team.

Figure 2.

Example simulated patient scenario mandating emergency assessment, ECG, and blood tests, referral for cardiac stress testing, and final discharge to a general practitioner.

Results

The pathway was developed within the existing EHR using the existing EHR provider’s business process management–based toolkit for configuring electronic, patient-centric, clinical decision workflows. A dynamic workflow document was created that included a collaborative record-keeping space, decision support elements in the form of both checklists and Boolean choices, risk score calculation, integrated messaging for test requests, and export to summary documentation and provided a single workspace for the whole patient journey from arrival in an ED, via outpatient clinic, to discharge to a community provider. Upon completion of the pilot development, the technical requirements had been met.

In total, 10 physicians participated in the think-aloud testing. All had prior experience with the hospital EHR but were new to the pilot pathway. Key critical barriers consistently identified by participants were (1) user experience and (2) the flexibility to align with real human workflows (Figure 3). Other important themes were change of work practice, styling, language and formatting, and clinical content. These themes were prioritized for their importance as barriers to implementation.

Figure 3.

Key themes from think-aloud testing, prioritized as potential barriers to implementation.

User experience feedback included a frustration with the click-through times between different sections of the pathway and not being clear where in the pathway they were at any given point.

Flexibility concerns included a desire to go back in a pathway, enter information in a different order, or have the ability to edit previously recorded information. For example, users asked, “what if I want to pause, order a CT scan, and then go back in the pathway?” and “what if my information about the patient changes during the consultation?”

This lack of flexibility was perceived as a critical risk by the clinical team. The risk identified was that of patient data changing (by virtue of changing patient condition or acquisition of new information), an edit being made to an earlier part of the pathway but the new information not propagating fully through the pathway to change the clinical management recommendations. Therefore, a clinical recommendation that was previously correct would now become incorrect or vice versa.

Given that a situation of changing patient status or dynamic information is common within emergency medicine, this risk was deemed fatal to this digital pathway pilot, and hence, project implementation was stopped.

Additional feedback deemed important to address but non-critical to implementation included clinical questions falling within the remit of the clinical governance team, language and styling issues, and those relating to changes in work practice. These issues were to have been addressed and implemented using further iteration and change management.

Discussion

This project demonstrated the necessity of usability testing prior to deployment of a new clinical tool. Despite successful technical transcription of a clinical pathway into the digital space, think-aloud testing of a new digital chest pain pathway identified critical barriers to implementation, namely, user experience, and rigidity of clinical workflow with the associated risk of decision errors.

The think-aloud feedback about user experience demonstrated that a frequent user concern was the inability to ascertain where in the pathway the user was at a given moment. It is notable that this concern is among the core principles of usability practice. ¹⁶ The perspective of two of the authors as practicing emergency medicine specialists is that user experience has not been prioritized in many health informatics systems or comes at the price of fulfilling technical requirements in a limited resource setting. A “Diderot effect” of changing expectations may also be at play where the ubiquity of intuitive and highly usable technology outside the work environment may be changing practitioners’ expectations of their work systems.

The think-aloud feedback about flexibility and workflow within the digital clinical pathway reflected real-world scenarios that could not be managed using the unidirectional and constrained pathway design. While linear or predictable pathways may be well suited to predictable sequences of care in outpatient clinics, it is perhaps unsurprising that simple transcription of a pathway into the digital space did not suffice in the realm of emergency medical care. Existing clinical pathways tend to be two-dimensional depictions of more complex human workflows. Clinical pathways can be eloquently written and visually depicted as a controlled linear sequence of information acquisition, decisions, and actions, however, our experience as practicing specialist clinicians is that the execution of real clinical work is more complex and may involve different entry points to an algorithm, cognitive task, and physical task that do not coincide in time or space, variable task ordering, information that can vary over time, and clinical choices that vary from those recommended.

These factors amount to a difference between the conceptual work that is imagined and published in clinical pathways and the work that is actually done in the clinical arena. While some may support a top-down approach to safety and systems design that would impose a new digital pathway upon clinicians in order to more tightly specify and constrain their clinical work and “fit the rules,” we believe that a more convincing alternative approach to safety in healthcare acknowledges that preserving real human behavior and flexibility in clinical work is essential to a resilient health system, ¹⁷ and therefore, digital pathways should be tailored to meet this requirement and “fit the work.”

Similar problems of conceiving clinical records in isolation from practical workflow have affected EHR systems in general, and despite almost universal adoption of digital systems, clinical benefits have been harder to identify than process improvements. ¹⁸ There are many possible reasons for this disappointing impact on patient care. Two of the most important are (1) that the EHR as a concept may be too generic, describing a diversity of systems with many and varied functionalities ¹⁹ and (2) that EHRs have historically been built more around the needs of the organization than the needs of the patient. Consequently, the clinical record has been called a “200 year-old 21st century challenge.” ²⁰

In emergency medicine, benefits of electronic notes have not been reliably demonstrated, ²¹ and unintended adverse consequences such as longer charting times, workaround development, and adverse social effects have been documented ²² with some suggestions that poorly designed systems are contributing to provider burn-out. A study of ED physicians in a community hospital ED showed that they spend significantly more time entering data into electronic medical records than on any other activity, including direct patient care. ²³ It is imperative that this usability concern is addressed so that emergency physicians can devote more time to direct patient care.

We propose that a useful and unifying theme to guide future digital pathway design in emergency medicine is closer alignment with physical and cognitive aspects of real clinical workflow. Unless decision tools operate seamlessly and flexibly with physical and mental tasks such as writing notes, communicating with other providers, ordering tests, and effecting treatments, their appeal to the clinician is likely to have a ceiling effect as even if they increase decision accuracy, they are unlikely to make actual clinical work easier. Further developments of digital pathways in emergency medical care will require more a flexible solution fit for this purpose (Figure 4).

Figure 4.

Key recommendations for digital pathway design in emergency medicine.

Conclusion

Usability testing revealed that despite technical success in transcribing an acute care pathway into the digital realm, factors such as user experience, flexibility, and clinical workflow were critical barriers to implementation into a live clinical environment. Emergency medical care has particular challenges that will require decision support solutions that both are intuitive to use and enable real cognitive and physical patterns of behavior, rather than constraining clinicians to an imposed sequence of actions.