Sage Journals: Discover world-class research

Abstract

In caring for patients with sepsis, the current structure of electronic health record systems allows clinical providers access to raw patient data without imputation of its significance. There are a wide range of sepsis alerts in clinical care that act as clinical decision support tools to assist in early recognition of sepsis; however, there are serious shortcomings in existing health information technology for alerting providers in a meaningful way. Little work has been done to evaluate and assess existing alerts using implementation and process outcomes associated with health information technology displays, specifically evaluating clinician preference and performance. We developed graphical model displays of two popular sepsis scoring systems, quick Sepsis Related Organ Failure Assessment and Predisposition, Infection, Response, Organ Failure, using human factors principles grounded in user-centered and interaction design. Models will be evaluated in a larger research effort to optimize alert design to improve the collective awareness of high-risk populations and develop a relevant point-of-care clinical decision support system for sepsis.

Keywords

clinical decision support human factors sepsis usability user-centered design

Introduction

Sepsis is a spectrum of organ dysfunction induced by infection¹ with increasing incidence^2,3 accounting for a total of US$20 billion in US hospital costs in 2011.^4,5 Current public perception of sepsis is low, as 88 percent of Americans have never heard of sepsis,⁶ even with a high estimated mortality rate of 25–30 percent.⁷ Multiple scoring systems have been derived and validated with the purpose of quantifying illness severity associated with sepsis and the probability of various outcome measures. Standardized in 1991, the original conceptualization of sepsis hinged on two of the four Systemic Inflammatory Response Syndrome (SIRS) criteria. These definitions, focused solely on inflammatory excess, were challenged due to a lack of specificity and clinical utility.^8–10 The Sepsis Definitions Task Force’s current definition of sepsis is no longer based on SIRS criteria, but instead based entirely on the objective existence of acute organ dysfunction as a downstream marker for the (mal)adaptive host response to infection.⁵ The evolution of risk stratification measures provides opportunities to further evaluate quantified organ system dysfunction as a quality improvement strategy for early sepsis detection. This research evaluates the utility of alert design for two sepsis staging models: quick Sepsis Related Organ Failure Assessment (qSOFA)⁵ and Predisposition, Infection, Response, Organ Failure (PIRO).¹¹

Clinicians are in need of improved electronic health record (EHR) alerts as clinical decision support (CDS) tools¹² to assist in early recognition of sepsis to combat high mortality rates.⁵ Time pressure, competing demands, and ambiguous alert design reduce a user’s opportunity to detect signals in the face of “noise” and may lead to inadvertent confirmation bias. As a result, CDS developers, designers, and users aspire to improve alert management to achieve better acceptance rates and improved care delivery. Currently, there are a wide range of alerts for sepsis in automated electronic sepsis alert systems within various clinical settings,^13–21 including alerts that integrate organ failure assessment scores and general severity scores;²² however, there remains a lack of evaluation and assessment of these alerts, specifically on clinical preference.^23,24

Compiling patient demographic and health information in a medium that affords rapid and accurate analysis to identify tailored treatment solutions, and delivering this information to the clinicians at the point of care can transform treatment. Unfortunately, many EHRs suffer from poor usability—providers struggle to use these systems efficiently, effectively, and safely. In response, we proposed to develop a framework to optimize EHR alert design to target sepsis²⁵ in line with human factors research best practice recommendations for the design of clinical reminders.²⁶ To determine how to best provide a sepsis alert to facilitate decision-making in the healthcare environment, we assessed the interaction of clinicians with health information technology (health IT) and the impact of CDS by controlling two design aspects: (1) how clinical information is displayed and (2) the level of content provided.

Methods

We developed four graphical model displays using two visual display types (text display and figure display) and two levels of content where level of content refers to the number of components required to calculate the score (PIRO representing high level and qSOFA representing low level). Display model prototypes were developed in Pencil, an open-source prototyping software (Evolus Co., LLC, Newport Beach, CA, USA). In the design of these displays, we considered the requirements of users performing the task and evaluated the impact of cognitive biases, usability, human factors principles, and user-centered design. All designs were created and intended for use by hospital staff, specifically on the general medical and surgical floor.

Results

The Figure Display Alert was designed using an adapted treemap to visually display a large amount of hierarchical data in a rectangular shape.²⁷ Treemap is a type of visual display described as a space-constrained visualization of hierarchical structures. This design is able to correlate a square’s size to possible point value, while still being able to indicate missingness, individual data elements, and raw values. The Text Display Alert was developed to appear similar to initial presentations of the original PIRO¹¹ and qSOFA⁵ depictions. Both alerts will display an integration of a sepsis staging system with associated mortality rates. For both figure and text display, following general design considerations were used:

Physical organization. A convention for informational alert displays in healthcare is to frame the key clinical details with two blocks of standard information: patient demographics on top and a summary message with context at an appropriate level of severity at the bottom.^28,29 It is particularly important to place routinely seen critical information such as patient name and triage admission time at the top of the alert where users can orient and re-orient to it easily. Negative space of the display should be taken into consideration of the display, to ensure users can digest information in manageable chunks without excessively separating related information. The use of negative space will also assist users who may be confused by physical separation of concepts, especially within the individual components of PIRO.

Visual cues and consistency. Fonts, colors, and placement of words that indicate meaningful outcomes need to be identical across displays for faster understanding.³⁰ Alerts should be displayed similarly, framed with a top and bottom box, and both displays (text and figure) require the same information.

Font size and typeface weight. Typeface weight is the thickness of the letters that can be expressed normal, bold, and so on. Font size and typeface width should be used in a clear and simple hierarchy to indicate the organization of content and reinforce differences in salience. Text at the same hierarchical level is recommended to have consistent size and typeface weight. Font should be appropriately sized for users to read a display and its components,^29,30 and typeface width should be consistent throughout displays.

Color. A limited, clearly distinguishable palette of colors is recommended to fill space in a graphic. Both text and fill colors should be selected to establish good contrast, facilitating readability of text.²⁹ It is particularly important to select salient, standard-following and consistent colors when indicating alarm.

Three-dimensional (3D) effects. In development of the displays, and in consideration of the user, 3D effects were considered for texture for color-blind users, as well as to help users interpret when elements have been reached.²⁸

Text displays

The text display for PIRO (Figure 1(a)) provides a list of important variables that could be applied at the bedside,¹¹ based on a total of 33-point scale. Each part of the PIRO acronym contains a different list of variables within the “P score,” “I score,” “R score,” and “O score.” qSOFA (Figure 1(b)) is based on a total of three points: systolic blood pressure, altered mentation, and respiratory rate. Variables are displayed in no prioritization to severity. The display layout ensured that the most important information was placed in a prominent position to ensure it is seen^31–34 and consistent with left to right reading comprehension in the English language.^35–37

Figure 1.

(a) PIRO text display and (b) qSOFA text display.

Physical orientation. P, I, R, O components are displayed in separate boxes under their acronym heading, with small negative spaces between each box to allow separate consideration of each heading. For qSOFA, components are presented in a single group, since separating three components would needlessly dissipate user attention. Critical information and total score and associated in-hospital mortality are at the top and bottom to frame the displays.

Font size. P, I, R, O labels are the largest to help users orient themselves. Summing scores are the next largest, emphasizing the scoring results for each heading and total score. Detailed scoring criteria are smallest, though still of legible size, cueing users to understand the availability and role of this information. The users are left with the task of understanding and prioritizing on text-based forms and rubrics.

Color. Light gray was selected for ease of text visibility within the text model, limiting the color palette to background, one fill color, and one text color.

Figure displays

The figure display (Figure 2(a) and (b)) presents the same information as the text display, illustrating each score component by letter. The alert is coded using colors, helping clinicians recognize patterns and exceptions, and making it quick and easy to identify if a patient’s condition has exceeded the associated thresholds. Over time, the clinician may associate certain patterns of the figure display alert with deterioration, using “at-a-glance” interface design.^34,38 The treemap design is strictly proportioned to match the score weight of each variable contributing to the calculation and does not contribute to misleading comparisons that may be caused by bar graphing score dimensions of unequal weights.

Figure 2.

(a) PIRO figure display and (b) qSOFA figure display.

Physical orientation. This figure display follows the conventional vertical progression, compact for screen economy, but also has physical separation of concepts so that users may focus on one aspect at a time.^29,38

Visual cues and consistency. All design aspects are standardized to help users perceive the designs between the text and figure. 3D effects were not included in figure display as they interfered with ability to perceive area in the treemap.³⁹

Font size. Typeface weights were consistent and clearly legible. P, I, R, O labels are the largest to help users orient themselves. Granular data and summing scores are of comparable size to allow users to pay attention to detail, the overall summary, or both according to preference. Health data labels are smallest as they are the most predictable and unchanging aspect of the display.^29,30

Color. A dark red was used to indicate potentially alarming data that give cause for concern regarding sepsis diagnosis. In the figure display, this allows the user a glancing assessment of threat seriousness.^29,30,40 Three colors were used: dark red for components that reached the threshold, muted blue for data that did not meet threshold, and gray to differentiate the patient demographics and summary score boxes. Missing information deserved special user consideration because it signifies incomplete diagnostic information, but not too much emphasis, which could detract from the display’s overall message. Missing data are shown with a fill color, the same color as the background, to indicate that it neither meets nor does not meet the threshold, and contains stripes to facilitate users’ perception that this information might be potentially concerning. Green color for non-threshold crossing components, considered during development, was not selected because it implies “safe” for users.⁴¹

Discussion

Human factors principles, standards, and guidelines provided considerations for the design of text and figure displays. Traditionally, alerts are system components that serve to direct a user’s attention to information related to some parameter which has been exceeded.⁴² Newer alerts, however, have advanced to the point of becoming a “type of automation that supplements the human powers of observation and decision.”³⁹ Alerts amplify the capacity of clinicians to continuously monitor changes in patient status and thereby support timely intervention. The purpose of an alert is to prompt an operator action, but poor alert system design has been a contributing cause of adverse events in numerous healthcare systems worldwide. The appeal of access to a large amount of clinical data must be balanced against the real possibility of information overload. Research demonstrates that medical displays are often incompatible with practitioners’ workflow and unnecessarily fragment patient information.⁴³ Information is often spread across multiple tabs and locations that require piecemeal information search and acquisition. This may confound practitioners’ ability to detect evolving changes, make it more difficult to attain a holistic view of a patient’s health state, lead to care inefficiencies, and frustrate clinicians. Recognizing limits on human working memory, clinicians need external automated information systems to support early detection of patient deterioration and improve timeliness of therapeutic response.

The timeliness and efficacy of clinical interventions may be enhanced with improved alert displays. Alert and warning complexity is especially prevalent in health IT. Despite this issue, there is little consensus on how alerts should be generated and displayed to the user^41,44 as well as what level of interaction is appropriate. Future research will evaluate these four alert displays using realistic patient data in an EHR test environment. Novel analysis of the association between provider-, system-, and patient-level data on provider performance and preference will provide evidence-based principles and actionable design recommendations to develop a relevant point-of-care CDS system for sepsis. Usability testing in simulation using test patients and real cases will determine the value of each display model in the context of sepsis severity based on clinician response.

The intent of this work was to develop the models that will be used in future work to solicit performance and preference from clinicians. In this planned study, we are comparing two sepsis severity scores selected as a starting point based on the level of information provided. This could be expanded in the future to evaluate the impact of different severity scores. The design aspects, how clinical information is displayed and the level of content provided, are independent of additional considerations like why score elements are missing and how to measure the impact of missing information on display design. Finally, our work does not consider the impact of the complex clinical environment or unique clinical environments (emergency department versus intensive care unit) on display design and clinician response. These complexities and variances may impact user preference and graphic display design requirements. The visual display models discussed in this study are tailored to clinicians with the goal to better equip clinicians to understand their patients’ data through feature extraction and visualization embedded with different asynchronous alerts. Future studies include considering the impact of clinical environment on display design, and comparing additional scores besides PIRO and qSOFA, to alleviate the limitations of this study.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Kristen E Miller

References

Seymour

Liu

Iwashyna

, et al. Assessment of clinical criteria for sepsis: for the third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA 2016; 315(8): 762–774.

Iwashyna

Cooke

Wunsch

, et al. Population burden of long-term survivorship after severe sepsis in older Americans. J Am Geriatr Soc 2012; 60(6): 1070–1077.

Gaieski

Edwards

Kallan

, et al. Benchmarking the incidence and mortality of severe sepsis in the United States. Crit Care Med 2013; 41(5): 1167–1174.

Torio

Andrews

National inpatient hospital costs: the most expensive conditions by payer, 2011. Statistical Brief #160. Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Rockville, MD: U.S. Agency for Healthcare Research and Quality, 2013.

Singer

Deutschman

Seymour

, et al. The third international consensus definitions for Sepsis and Septic Shock (Sepsis-3). JAMA 2016; 315(8): 801–810.

Rubulotta

Marshall

Ramsay

, et al. Predisposition, insult/infection, response, and organ dysfunction: a new model for staging severe sepsis. Crit Care Med 2009; 37(4): 1329–1335.

Kumar

Taneja

, et al. Nationwide trends of severe sepsis in the 21st century (2000-2007). Chest 2011; 140(5): 1223–1231.

Vincent

Dear SIRS, I’m sorry to say that I don’t like you. Crit Care Med 1997; 25(2): 372–374.

Kaukonen

K-M

Bailey

Pilcher

, et al. Systemic Inflammatory Response Syndrome criteria in defining severe sepsis. N Engl J Med 2015; 372(17): 1629–1638.

10.

Levy

Fink

Marshall

, et al. 2001 SCCM/ESICM/ACCP/ATS/SIS international sepsis definitions conference. Crit Care Med 2003; 31: 1250–1256.

11.

Howell

Talmor

Schuetz

, et al. Proof of principle: the predisposition, infection, response, organ failure sepsis staging system. Crit Care Med 2011; 39(2): 322–327.

12.

Amland

Hahn-Cover

KE.

Clinical decision support for early recognition of sepsis. Am J Med Qual 2016; 31: 103–110.

13.

Makam

Nguyen

Auerbach

AD.

Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: a systematic review. J Hosp Med 2015; 10(6): 396–402.

14.

Hooper

Weavind

Wheeler

, et al. Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit. Crit Care Med 2012; 40(7): 2096–2101.

15.

Meurer

Smith

Losman

, et al. Real-time identification of serious infection in geriatric patients using clinical information system surveillance. J Am Geriatr Soc 2009; 57(1): 40–45.

16.

Nelson

Smith

Jared

, et al. Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med 2011; 57(5): 500–504.

17.

Nguyen

Mwakalindile

Booth

, et al. Automated electronic medical record sepsis detection in the emergency department. PeerJ 2014; 2: e343.

18.

Thiel

Rosini

Shannon

, et al. Early prediction of septic shock in hospitalized patients. J Hosp Med 2010; 5(1): 19–25.

19.

Berger

Birnbaum

Bijur

, et al. A computerized alert screening for severe sepsis in emergency department patients increases lactate testing but does not improve inpatient mortality. Appl Clin Inform 2010; 1(4): 394–407.

20.

McRee

Thanavaro

Moore

, et al. The impact of an electronic medical record surveillance program on outcomes for patients with sepsis. Heart Lung 2014; 43(6): 546–549.

21.

Sawyer

Deal

Labelle

, et al. Implementation of a real-time computerized sepsis alert in nonintensive care unit patients. Crit Care Med 2011; 39(3): 469–473.

22.

Horton

Graves

Building a sepsis alert system. J Hosp Med 2016; 11(Suppl. 1): 1–10.

23.

Dziadzko

Harrison

Tiong

, et al. Testing modes of computerized sepsis alert notification delivery systems. BMC Med Inform Decis Mak 2016; 16(1): 156.

24.

Miller

Kowalski

Capan

, et al. Assessment of nursing response to a real-time alerting tool for sepsis: a provider survey. Am J Hosp Med 2017; 1(3): 1–10.

25.

Miller

Capan

, et al. Operationalizing sepsis alert design and clinical decision support: developing enhanced visual display models. Proc Int Symp Hum Factors Ergon Heal Care 2015; 4(1): 103–109.

26.

Saleem

Patterson

Militello

, et al. Impact of clinical reminder redesign on learnability, efficiency, usability, and workload for ambulatory clinic nurses. J Am Med Inform Assoc 2007; 14(5): 632–640.

27.

Kong

Heer

Agrawala

Perceptual guidelines for creating rectangular treemaps. IEEE Trans Vis Comput Graph 2010; 16(6): 990–998.

28.

Phansalkar

Zachariah

Seidling

, et al. Evaluation of medication alerts in electronic health records for compliance with human factors principles. J Am Med Inform Assoc 2014; 21(e2): e332–e340.

29.

Horsky

Schiff

Johnston

, et al. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. J Biomed Inform 2012; 45(6): 1202–1216.

30.

Khajouei

Jaspers

MWM

. CPOE system design aspects and their qualitative effect on usability. Stud Health Technol Inform 2008; 136: 309–314, http://www.ncbi.nlm.nih.gov/pubmed/18487749

31.

Khajouei

Jaspers

MW.

The impact of CPOE medication systems’ design aspects on usability, workflow and medication orders a systematic review. Methods Inf Med 2010; 49(1): 3–19.

32.

Kawamoto

Houlihan

Balas

, et al. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330(7494): 765.

33.

Tsopra

Jais

J-P

Venot

, et al. Comparison of two kinds of interface, based on guided navigation or usability principles, for improving the adoption of computerized decision support systems: application to the prescription of antibiotics. J Am Med Inform Assoc 2014; 21(e1): e107–e116.

34.

Linder

Schnipper

Tsurikova

, et al. Documentation-based clinical decision support to improve antibiotic prescribing for acute respiratory infections in primary care: a cluster randomised controlled trial. Inform Prim Care 2009; 17(4): 231–240.

35.

Maviglia

Zielstorff

Paterno

, et al. Automating complex guidelines for chronic disease: lessons learned. J Am Med Inform Assoc 2003; 10(2): 154–165.

36.

Shiffman

Liaw

Brandt

, et al. Computer-based guideline implementation systems. J Am Med Inform Assoc 1999; 6(2): 104–114.

37.

Tsopra

Lamy

Venot

, et al. Design of an original interface that facilitates the use of clinical practice guidelines of infection by physicians in primary care. Stud Health Technol Inform 2012; 180: 93–97.

38.

Belden

Patel

Lowrance

, et al. Inspired EhRs designing for clinicians (ed. The Curators of the University of Missouri), 2014, http://inspiredehrs.org/Inspired_EHRs_Designing_for_Clinicians.pdf

39.

Meyer

Flat design: its origins, its problems, and why flat 2.0 is better for users. NN/g Nielsen Norman Group, 2015, https://www.nngroup.com/articles/flat-design/

40.

Russ

Zillich

McManus

, et al. A human factors investigation of medication alerts: barriers to prescriber decision-making and clinical workflow. AMIA Annu Symp Proc 2009; 2009: 548–552.

41.

Phansalkar

Edworthy

Hellier

, et al. A review of human factors principles for the design and implementation of medication safety alerts in clinical information systems. J Am Med Inform Assoc 2010; 17(5): 493–501.

42.

Pritchett

Vándor

Edwards

Testing and implementing cockpit alerting systems. Reliab Eng Syst Saf 2002; 75(2): 193–206.

43.

Koppel

Kreda

DA.

Healthcare IT usability and suitability for clinical needs: challenges of design, workflow, and contractual relations. Stud Health Technol Inform 2010; 157: 7–14.

44.

Ash

Sittig

Campbell

, et al. Some unintended consequences of clinical decision support systems. AMIA Annu Symp Proc 2007; 2007: 26–30.

Informatics and interaction: Applying human factors principles to optimize the design of clinical decision support for sepsis