Assessing Clinician Experience: Contributions from Human Factors

Abstract

Evidence points to the critical importance of creating a positive clinical experience to maintain a well-functioning healthcare system. Despite its importance, there is little guidance on how to assess clinical user experience. While there does not yet appear to be consensus on critical domains of clinical experience, most components that appear in the literature or as categories in proprietary descriptions include areas in which human factors can and should have an impact. Panelists will discuss their experiences with various approaches to assessing clinical user experience and associated advantages and disadvantages. Specific approaches discussed include how technical and system data can be triangulated with other data, use of MITRE’s Resilience-Aware Development structured observation form (RAD-O), application of various survey strategies, and application of a work-centered approach to evaluation. A clinician also brings the perspective of the impact of using multiple electronic health records simultaneously on user experience.

Introduction

Even before the COVID-19 pandemic, discussions around clinician wellbeing were beginning to acknowledge burnout’s implications and the need to improve health care worker (HCW) experience and satisfaction. A recent report projects shortages in all levels of the healthcare workforce by 2026 (Bateman et al., 2021). Operationally, many consulting firms offer survey services to measure employee satisfaction or employee experience, as proposed by Hunt et al. (2023). Even with specific details being proprietary, the list of assessment categories that are often included is quite long and includes many areas in which human factors (HF) projects can and should be having an impact. One inherent risk of surveys is that response burden could add to burnout; however, a recent review provides a framework with factors contributing to the perception of response burden across a survey cycle so survey developers can minimize the burden (Yan & Williams, 2022).

In terms of HCW experience and satisfaction, these questions remain outstanding:

• How can organizations assess employee experience without contributing to burnout in doing so?

• How should organizations assess employee experience to maximize the relevant factors captured?

• What can HF contribute to conducting assessments in our work?

• How can HF enrich the assessment toolbox?

Our panelists explore these questions, citing experience specifically with:

• Integrating multiple methods to understand real-time EHR use as a work tool by front-line clinicians

• Triangulating technical and system data with multidisciplinary site visits

• Applying generic and domain-specific evaluation measures from the HF literature to a multi-hospital setting with multiple software applications used at once

• Using traditional HF methods in combination with work-centered evaluations to provide diagnostic feedback

• Identifying capabilities of work systems that enable work system resilience to challenging situations.

Shawna Perry, University of Florida Jacksonville

Have HCWs Become Tools of Their Tools?: The Electronic Health Record (EHR) and Clinical “Work-as-Done”

The introduction of information technology (IT)-based tools into health care delivery has proven to be as problematic as predicted (Berg, 2001, Guo et al., 2017) but unfortunately poorly attended to until after this domain reached its breaking point during the COVID-19 pandemic. This global catastrophe sapped already frayed cognitive, psychological, emotional, and physical reserves being tapped into for almost 20 years pre-pandemic by health care workers at the frontline of care coping with escalating technology-based demands. The EHR is a chief contributor to health care worker (HCW) burnout and job dissatisfaction (Ehrenfeld & Wanderer, 2018; Gardner et al., 2019), with time and effort previously directed toward the meaningful provision of care through hands-on human-to-human interactions, now being supplanted by onerous and tedious interactions with the EMR. As one primary care physician stated, “I am no longer a physician but the data manager, data entry clerk, and steno [person]. . . I became a doctor to take care of patients. I have become the typist” (Bodenheimer & Sinsky, 2014).

Attempts to elucidate how the EHR contributes to “technostress” for its users, which is stress experienced by end users in organizations as a result of their use of information and communication technology (Ragu et al., 2008), are rarely helpful to those “trying to make it work” at the frontline of care. This is because the majority of these studies are from the stance of EHR/software designers who tend to think of “usability” as it relates to the performance of the software and not how the EHR as a clinical “tool” integrates into the dynamic nature of clinical work “in the wild.” As a result, opportunities for substantive EHR changes are ineffective and instead create new issues that compound provider dissatisfaction. This ultimately erodes trust in the EHR, including aspects that contribute to sensemaking, risk mitigation, and patient safety in effective ways.

The EHR has become an irrevocably engrafted clinical tool. As such, it is imperative to identify meaningful EHR design changes that reduce its harmful effects on well-being and professional longevity. The nature of clinical work, no matter the setting (e.g., hospital, home, clinic), is delivered within a sociotechnical work system. Studying any aspect will require multiple approaches—including studying information technology tools (Berg, 1999). Given this premise, research on the EHR should include the following approaches for clarity of “real-time” EHR use as a work tool: (a) obtaining and studying stories from HCWs of critical incidents or memorable patient care cases related to the EHR, (b) capturing quantitative data on how the HCW is directly interacting with the EHR during the course of their clinical work, and (c) direct observations within the “live” clinical setting to understand performance shaping factors and incongruent workflows that impact user experience with the EHR.

The grouping of these approaches would provide more nuanced foundations from which to design/redesign to reduce technostress and HCW cognitive overload in this highly consequential domain. HF, as a discipline, is uniquely positioned to meet this challenge.

Shawna Perry is an Associate Professor in the Department of Emergency Medicine at the University of Florida Jacksonville. She is an internationally recognized expert in patient safety and human factors engineering in healthcare. She has deep experience in clinical medicine as a practicing Emergency Medicine physician on the frontline and in hospital leadership roles.

Kyle Maddox, Department of Veterans Affairs

Department of Veterans Affairs Efforts to Assess User Experience (UX)

The Department of Veterans Affairs (VA), Veterans Health Administration (VHA) is the largest integrated health care organization in the United States, with more than 9 million Veterans enrolled. VHA employs over 371,000 health care professionals across 1,321 health care facilities. Health information technology (HIT) system usability and technical performance directly impact the efficiency and satisfaction of health care employees. They can provide informative insights on the quality and safety of health care provided to patients (Tutty et al., 2019). Since 2018, VHA has been utilizing two EHRs, and VHA’s HF professionals have been supporting both systems. In this complex environment, with so many critical stakeholders, HF and UX are foundational to VHA success.

HF expertise is embedded alongside Clinical Informatics teams in the Digital Health Office (DHO), a national program office within VHA. This allows HF experts to cross silos and understand issues experienced by end users. HF professionals are then able to advocate for the clinical end user. VHA HF teams participate in various efforts, including initiatives to develop style guides and design templates, heuristic evaluations of other templates, time-motion studies to compare workflows, cognitive task analysis for ordering prosthetics, training needs analysis, artificial intelligence (AI) experimentation, and more.

An example of efforts to assess and improve the user experience with VHA’s new Federal EHR is VHA’s End User Experience Clinics, which are multidisciplinary site visits facilitated by the Human Systems Integration (HSI) team within DHO. The End User Experience Clinic is based on a modified contextual inquiry (Beyer & Holtzblatt, 1999) and is used to understand and resolve system performance and UX issues for EHR users at multiple VHA facilities. A mixed team, led by HSI’s HF engineers, conducts the clinic at a site. It includes systems architects, local and national informatics, a team from the commercial EHR vendor, and (information technology) IT professionals who work to provide at-the-elbow support to the end user. Interview data is thematically analyzed. Findings are shared in a summary report and via UX maps, such as service blueprints or journey maps, to visualize pain points and opportunities for VHA leadership.

Other efforts to assess UX for the new Federal EHR include:

• Focus Groups with cohorts of participants, based on role and representing different facilities, are utilized to discover and understand system performance issues that impact UX, brainstorm solutions, review designs or workflows, and elicit requirements.

• Surveys measure end-user sentiment, understanding of interventions, and general satisfaction. These are administered at different times and include commercial items, validated items in the public domain, and questions unique to interventions.

• Telemetry tools measure system performance and user experience key performance indicators (KPI). Examples of important KPIs include user interruptions, incident and outage free time, round trip time, and time spent charting after hours.

• Issue Resolution and Tip Sheets are used to complete communication efforts back to study participants and share interventions to end users across the Enterprise.

• Baseline Usability Testing before and after implementation of the new Federal EHR allows VHA to assess how users and facilities are progressing toward their baseline UX performance.

Kyle Maddox is an Engineering Psychologist with the VA’s (HSI) division in the Office of Clinical Informatics. He is acting Program Manager for HSl’s support of the Electronic Health Record Modernization, which includes an EHR transition. He also leads a team focused on improving Federal EHR users’ technical experience.

Emily Patterson, The Ohio State University

Generic versus Domain-Specific UX Measures in Healthcare

HF concepts relevant to “generic” (i.e., domain- and technology-independent) UX have been developed, validated, and extensively used. Examples include the System Usability Scale (SUS) for usability (Lewis, 1995), the task completion time for efficiency, and the NASA Task Load Index (TLX) (Hart & Staveland, 1988) for mental demand. An emerging generic measure for trust in a system with embedded automation is the System Trustworthiness Scale (STS) (Alarcon et al., 2024). An important benefit of these measures is that it is possible to use or assign justifiable quality thresholds for advancing technology to the next stage of the development process, supporting benchmarking across applications and tracking trends over time and after interventions.

The International Organization for Standardization (ISO) defines UX as “the user’s perceptions and responses that result from the use and/or anticipated use of a system, product or service” (ISO/TR 25060:2023).

In a hospital environment with a mature digital infrastructure, clinician UX is more complex and thus more complicated to assess and measure. For example, clinicians in roles with responsibilities delineated by national and state licensing and accrediting bodies are supported by a suite of only partially integrated tools such as the EHR, access to historical data from other applications, and clinical decision support that has both generic and hospital-specific business rules guiding behavior. A clinician in a hospital setting likely has over 100 clinical and non-clinical applications that impact their clinician user experience, including communication technologies like EHR chat, pagers, hands-free communication devices, and email.

A broader concept than user experience is clinician experience, defined as “clinicians’ perceptions of the quality and safety of care provision, interprofessional collaboration, and work environment, their engagement in decision-making, and psychological experiences in the workplace.” (Pervaz Iqbal et al., 2020) Domain-specific measures include the Clinician Experience Measure (CEM-10) (Harrison et al., 2024) and the EHR Satisfaction Survey (Garabedian et al., 2023).

In all complex, sociotechnical domains with new technologies with embedded automation/AI, a new evaluation tool is the Resilience-Aware Development Critical Risks (MITRE RAD-CR^TM) checklist (Patterson & Neville, 2023a). This checklist supports monitoring and identifying critical risks involving the unintended loss of important capabilities for a work enterprise’s resilience when a change is introduced.

Emily Patterson is a Professor in the School of Health and Rehabilitation Sciences, College of Medicine, at The Ohio State University. She applies human factors to improve patient safety in healthcare with research and operational efforts. She is an associate editor for the Human Factors in Healthcare journal. She directs a certificate in Usability and User Experience in Health Care.

Emilie Roth, Roth Cognitive Engineering

Evaluating and Improving User Technical Experience

The optimistic promises for new health care IT systems often do not materialize because of a failure to recognize the complexities of the actual work context. This can lead to frustration, increased workload, and a need for workarounds. A recent example is an artificial intelligence system to detect diabetic retinopathy (Beede et al., 2020). While it performed at a “specialist” level under controlled laboratory conditions, it failed to live up to its promise once introduced in real-world clinics. The AI technology depended on uploading high-quality images impractical to generate under clinic conditions (e.g., poor lighting and slow internet speeds). Consequently, the AI system rejected 20% of the images uploaded. This resulted in increased stress and workload on the nurses, excessive processing delays, and unnecessary referrals to far away hospitals for patients.

Various HF tools are available to help anticipate, identify, prevent, and overcome barriers to the effective use of new technologies. These include field observations, practitioner interviews, and focus groups (Hettinger et al., 2019). The critical decision method is one of the most powerful (Klein et al., 1989). The critical decision method is used to elicit specific situations that users have experienced that provide a concrete illustration of sources of challenge and frustration, contributors to suboptimal performance and error, as well as strategies and workarounds that experienced practitioners have developed to overcome challenges and contribute to a more resilient sociotechnical system. A corpus of critical incidents provides an invaluable resource for identifying barriers to effective performance and contributors to poor user experience. This includes domain complexities that are likely to challenge the performance of any agent—human or technology-based.

Another effective tool for anticipating and mitigating poor user experience is work-centered evaluations of new technologies (Roth et al., 2021). Work-centered evaluations provide feedback on user technical expertise that can be used diagnostically to identify specific system characteristics that require improvement. Work-centered evaluations employ multiple measures that collectively assess the usefulness and usability of system design elements, as well as the ability of the system as a whole to meet the cognitive support objectives. Cognitive support objectives refer to the specific cognitive and collaborative performance benefits that the new technology (e.g., a decision-support system) is explicitly designed to provide. Usability refers to how easy each design feature is to learn, understand, and use. Usefulness refers to how useful a specific feature is from the perspective of supporting users in achieving their work objectives. Work-centered evaluations collect objective measures of performance using work-based scenarios that reflect the range of complexities that arise in the work domain, as well as user assessments of the new technology obtained via work-centered questionnaires.

A unique aspect of work-centered evaluations is the use of diagnostic questionnaires that are specifically tailored to the particular technology under evaluation. The objective is to identify which specific features of the technology contribute to a positive user experience and which need to be improved. Work-centered questionnaires combine closed-form rating questions and open-ended feedback fields to probe how effective the system is in meeting pre-identified cognitive support objectives and how useful and usable different system elements are. This approach to questionnaire design contrasts with more standardized user-assessment methods such as SUS, which are intended to be used in a summative manner and are explicitly not diagnostic.

Work-centered questionnaires also complement the collection of more free-form responses to user experience, such as verbal debriefs, focus groups, and free-text problem descriptions submitted to IT. Mean rating scores obtained on usability, usefulness, and cognitive support objective questions can provide concise summary statistics to present to customers, sponsors, and stakeholders as objective evidence of user experience that can be tracked over time.

Emilie Roth is the Principal Scientist of Roth Cognitive Engineering. Dr. Roth has conducted research and application in the area of cognitive performance and ways to enhance it for over 30 years. Areas of application include process control, medical systems, and railroad operations.

Kelly Neville, Mitre

User Experience Findings That Make a Difference

Once user experience has been assessed, how are the results used? Ideally, they lead to improvements: policies are improved, ambiguities and sources of potential coordination breakdown are addressed, technology designs are adapted, or other changes are made. However, faced with the often daunting and expensive prospect of researching, designing, iterating the design, and potentially adding the work to a contract, these improvements are often rationalized as unnecessary and the associated risks as acceptable. When technology development teams are faced with user experience data about risks that may require re-tooling of new technologies, it has been this author’s experience that many technology development teams begin to downplay the data. The common refrain is, “Every expert/operator has an opinion. Reacting to opinions is not good engineering.”

How do we get user experience data and the problems they reveal to be taken seriously and viewed as worthy of investment? This has been the focus of MITRE’s Resilience-Aware Development (RAD)^TM effort. The goal of this effort has been to identify characteristics and capabilities of work systems that enable work system resilience to challenging situations. The RAD effort targets high-consequence work systems for which mission failure can have grave and far-reaching consequences. As the number of resilience enablers in a work system decreases, the work system is at increasing risk of mission failure. In addition, we identified particularly fundamental resilience enablers and developed the MITRE RAD-CR^TM screening checklist. If one of the 20 critical risks in the checklist is missing or compromised, the work system is projected to struggle or fail to overcome challenges and accomplish its mission.

Other factors that limit the impact of user experience assessments include understanding the complex system dynamics associated with a user experience issue and its mitigation and translating mitigations into a form that is actionable for engineers or policymakers responsible for implementing the change. To address the first of these, user experience assessment tools such as RAD-CR and RAD surveys (available at trusts.mep.mitre.org) can be used with methods that provide a view into system dynamics. Before completing a user experience assessment, a field exercise, human-in-the-loop simulation, or expert interview is conducted to give analysts and work domain experts a fresh reference point for considering work system dynamics during challenging situations and involving proposed changes. These activities set the stage for responses and analyses that are better grounded in the realities of work-system dynamics.

One approach to translating user experience findings into actionable mitigations by engineers and policymakers assumes an iterative design process (e.g., Deal & Hoffman, 2010). Iteration, as used in Agile and DevOps approaches, permits the resilience impacts of work system changes to be iteratively considered and addressed, encouraging the evolution of the change based on an intermittent flow of findings and feedback. At the end of each iteration, analysts conduct lightweight user experience challenge activities, for example, 1-hr tabletop exercises (e.g., using the IDEAS-RAD Exercise Protocol [RAD-XP]^TM; Dorton et al., 2023), in situ or computer-based simulations, or expert what-iffing exercises (What if a surgery is running two hours longer than normal and now the closing surgeon is not available? How might things proceed if that happens and now you have New Capability Y?) featuring challenging situations. During challenge activities, observers capture resilience gaps and capabilities using the RAD-Observation (RAD-O)^TM data collection guide (Patterson & Neville, 2023b). The RAD-O guides capture and synthesis of observed work system dynamics that map to resilience enablers. During or after challenge activities, the RAD-CR checklist evaluates resilience gaps and capabilities for their criticality. Critical observation, checklist, and survey results for each resilience category are then written as user stories or in other formats developers and policymakers are accustomed to using.

Kelly Neville is a Principal Cognitive Systems Engineer at MITRE. Her work focuses on the dynamics of complex sociotechnical systems, enabling their resilience to challenges in high-stress and high-uncertainty conditions, and the role of expertise and work system design in achieving that resilience.

©2024 The MITRE Corporation. ALL RIGHTS RESERVED.

Conclusion

In conclusion, our panelists explore the application of HF principles to potential healthcare workforce shortages due to increased issues with clinician well-being and suboptimal clinical user experience with health information technology. The experiences of our panelists confirm that HF professionals are uniquely poised to help large, complex healthcare systems plan and conduct assessments, analyze the data, and plan sociotechnical systems changes based on the results. In addition, HF professionals can help communicate the advantages and disadvantages of various assessment methods and the timing of any data collection around interventions.