Demonstrating the Value and Impact of Human Factors-based Projects in Healthcare

Abstract

Institutionalizing human factors in healthcare organizations is a formidable enterprise. While much progress has been made over the years, challenging disconnects remain in applying and communicating about human factors tools and methodologies in the healthcare domain. Our objective was to explore how human factors professionals, especially those embedded within a healthcare organization, assess the value and impact of their healthcare operations-based projects. Toward this end, we interviewed eight human factors professionals working in healthcare to understand strategies that work well for them in demonstrating the value and impact of their work, as well as pitfalls to avoid. While especially relevant to the healthcare domain, this knowledge may help other human factors professionals in any industry develop strategies and plans to demonstrate value from their work, communicate about impact, and thereby grow the reach of human factors.

Introduction

Healthcare institutions can seem like an inscrutable realm for human factors professionals, especially those who are starting careers in this domain. When beginning to apply human factors methods and training in healthcare, one must first learn the language of healthcare and how this complex institution operates. Thus, applying and normalizing human factors in healthcare organizations can be a challenging undertaking. While substantial progress has been made, it often remains difficult to apply human factors methods and tools in healthcare domains. For example, Perry et al. (2021) argue that healthcare is a “variably opaque” institution with little flexibility and that human factors professionals must be willing to adapt to achieve greater integration within healthcare organizations. Others have proposed recommendations to improve integration of human factors in patient safety and healthcare, including building capacity among healthcare workers to understand human factors and increase the number of human factors practitioners in healthcare organizations (Gurses et al., 2011). Similarly, others argue that closer collaboration between clinical stakeholders and human factors professionals will result in substantial and beneficial changes to both professions and clinical care (Catchpole et al., 2021). However, misconceptions about human factors within the healthcare community has impeded progress (Russ et al., 2013).

Our objective was to understand how human factors professionals assess the value and impact of human factors-based operational projects, for example, non-research. Often, benefits such as improved usability or satisfaction are more difficult to measure than other components such as adoption. This objective also allows us to learn what works well for others, as well as pitfalls to avoid, when attempting to demonstrate the value and impact of human factors work within a healthcare organization.

We initially attempted to distinguish the meanings of “value” and “impact” during the conception of this project and design of the interview guide, where “value” was the outcomes that matter to clinicians and patients, and “impact” was more about any measurable differences the design/redesign a system or product may have after implementing it in a healthcare environment. However, we learned that our participants viewed these terms as mostly synonymous.

Methods

We planned to interview expert human factors healthcare professionals during the Human Factors and Ergonomics Society (HFES) 2023 Annual Meeting in Washington, D.C., October 23 to 27. We reviewed the conference program ahead of time for authors of accepted presentations for the healthcare track. We looked to include human factors professionals who were embedded within a healthcare organization as well as human factors professionals who owned their own consulting businesses and engaged with healthcare organizations as their livelihood depends on their ability to show value. Potential interviewees were contacted ahead of the conference to schedule 30 min interviews. Two participants from the same healthcare organization were scheduled for the interview after the conference via videoconference. All other participants were interviewed in person during the conference.

Participants

We interviewed eight human factors healthcare professionals (five females, three males). Six of the participants were embedded in four distinct, large healthcare organizations. The other two owned their own small consulting businesses and engaged in projects with various healthcare organizations. All participants had many years of experience in applied human factors projects in healthcare operations, as well as research-based projects. The interviews focused on the operational side of their work (i.e., funded within the healthcare organizations), rather than projects funded by external research bodies.

Semi-structured Interviews

The following questions were included in an interview guide:

• Does your organization use the terms “value” and “impact”? If so, in a similar or different way?

• Do you, and if so—How do you or your organization assess the impact of your human factors-based projects?

• About timing—when do you think about assessing the impact? Is that something you start assessing at start of the project? At the end?

• How would you describe the relationship between impact and value as they relate to your projects?

• How do you assess the value of your human-factors based projects, other than cost or cost savings? For example, do you use any value-based principles or metrics?

• What data and measures do you typically include, perhaps as a default? (e.g., burnout, cognitive load)

• How do you convince senior executives to buy into human factors?

• How do you conduct and utilize cost-benefit analyses in your organization?

• Assuming cost-based or activity-based accounting, is “human factors” a separate cost or embedded more broadly—that is, is there a line item for HF?

• Do you have recommendations for certain communication skills or strategies that could be used to advance the practice of human factors engineering within a healthcare organization?

However, given the semi-structured nature of the interviews, the questions served as prompts as needed; not all questions were asked of each participant. Rather, the discussion was focused on questions most relevant to each participant’s experiences. Each interview was audio-recorded and transcribed for analysis.

Analysis

Consistent with content analysis (e.g., Bengtsson, 2016), the interview responses were analyzed for the presence of content related to strategies or concepts for demonstrating the value and impact of human factors work in healthcare, as well as pitfalls and other relevant considerations. The lead author organized the interview responses around this guiding framework to generate categories for what works well, pitfalls, and other important considerations, with a co-author confirming or suggesting edits to each. This type of auditing procedure by a second analyst is considered an acceptable alternative to using independent analysts for ensuring validity of the analysis (Holden, 2010). We also used member checking to strengthen the validity of the findings (Creswell, 1994); a draft of the study findings was shared with each participant for their feedback to ensure the credibility of the results and resonance with their experiences.

Findings

What Works Well

Degree of Implementation

Several participants noted the importance of implementation of the process or system that they have worked on as a clear measure of success for their human factors work, although it might not be a binary measure. One participant noted: “Often our designs would not translate over to their systems and then it became more at the level of design requirements translating over [as a measure of impact/value].” And another mentioned: “Another way I think about it is that we often don’t get beyond a prototype stage, but if we have an application and an evaluation that suggests this would have a positive impact, that is of value to me.”

Publications and External Visibility

One way to raise the profile of human factors professionals within a healthcare organization is through publications and presentations at conferences. Although this is a standard job performance measure for researchers, some participants mentioned the importance of this type of external visibility for operations-based human factors healthcare work as well. An example quote was: “I do think publications matter in the healthcare world. It’s not enough to have done a really good study or to design something really good. But to have it peer-reviewed gives it way more credibility, sometimes more than it deserves, ha ha [sic]. So that is important to me and that is what I think our community values.” Another participant noted: “Now when we present on operations, that does lend to the quality of the work we do. Not everybody, but some leaders in the hospital value that. That we’re not just a team that doesn’t know what we’re doing. We are recognized not just locally but also nationally and internationally.”

Reputation and Relationship Building

Although external visibility can be helpful, participants also talked about the importance of internal visibility through reputation and building relationships within the healthcare organization. As one participant said: “These guys may not have read any of our papers or gone to our conferences. But they remember. . . .to the extent that you have internal groups that remember the last system you built and it was successful, then that makes it easier to have an entrée. . .to be brought in when the next system comes along.” Another participant emphasized: “A lot of it is just relationship building over time. We have to work less hard to demonstrate our impact because our impact has now been felt over multiple service lines across the system and so we’ve built up our credibility.”

Measures

Participants named specific measures that they use to demonstrate the value or impact of their human factors work, including financial impact, utilization, shortened time to complete tasks, and less errors. However, some participants noted that benefits may not be quantifiable or may be difficult to quantify. In these cases, some participants emphasized the importance of collecting both objective, quantifiable measures (when possible) along with qualitative data. One participant explained: “If you talk to six people you’re going to learn a lot, and you’re going to know how this is helping or not. But if you go and write about or talk about what you learned from six people, they’ll be like, ‘Hmm, this is interesting, but it’s six people.’ When you give people a very superficial survey and you get a lot of people to answer it, that feels compelling, so it’s a hard thing. And I’m more and more thinking about, how do you get some of both.”

Storytelling and Communicating Past Successes

Communicating a compelling story to healthcare stakeholders was considered an important skill by participants when demonstrating the value of human factors work. One participant provided a recent example: “. . .and we shared it [the alert design] with some clinicians and they said ‘Oh, if this patient is that high risk for an overdose, I’m going to refer them immediately to a pain clinic or a substance use disorder place,’ which is the opposite of the intent. The intent was to get these clinicians to give this patient more attention, as opposed to just abandoning them. And so the team was completely alarmed by this. We were able to redesign the alert so that the recommended actions were more salient. So, I think that’s a compelling story.” Storytelling can also include examples of past successes. For example, another participant explained: “In communicating with top executives and leadership, certainly the bottom line is very important. We want to point to those few projects where we can say look at the costs we saved the organization. I think it’s also demonstrating how we supported those organizational level goals or solved those system-wide challenges.”

Internal Champions

Having internal champions within the healthcare organization to advocate for the human factors work was identified as critical by some participants. This is a natural outcome of relationship building and having past successes. As one participant noted: “If you’re able to identify a few key members of leadership who are already bought in, or who you can easily buy in, then you start there.” Another participant explained: “Another important factor is that we have human factors champions within the organization. I think that is very valuable and they could also help us move forward with the human factors interventions.”

Being Proactive

Finally, being proactive was a trait described by one participant as an important strategy: “At this point a lot of people just ask for our help, but we’re looking at our system and we know what are high priorities for the organization and where we see an opportunity to add value, we jump in. We get in touch with the leaders of that project or friends of the leaders of that project. We start to offer what we can offer and we have been brought in and have gotten a lot of exposure to high-level organization-wide projects though doing that. Just being able to let our reputation speak for itself. But you kind of have to whittle your way in a little it.”

Pitfalls

Lack of Humility

Based on extensive experience, one participant warned against a lack of humility: “This may be a bit trivial, but I feel like one position in the human factors community is to talk about how terrible this design is, and now we’ve made it better. That’s often insulting to the people who did the original work. I’ve done designs that have felt so much better than what we’ve done before but then years later, they’re terrible again. So I’ve really changed the way that I tell stories to ‘here’s what we started with, it was a good start, and for these reasons—that’s why we did this, or they did this. And now here’s how the world changed and here’s what we learned and we need to think about this differently. And so I’m probably preaching to the choir but I think a little more humility in our storytelling—like everyone else is really stupid and they do these completely ridiculous things and we come in and we can see all these problems. . .I think that’s not the way to go anymore.”

Failing to Adapt Language That Will Connect With Your Audience

Several participants described an inability to adapt human factors language for others in the healthcare organization as a substantial pitfall. One participant explained: “. . .healthcare as an institution is going to be inflexible. And so it’s incumbent on human factors to say, ‘How do we weasel in to get them to see us through their lens, their nomenclature, their perspective?” Another participant noted: “I don’t mind using labels that are most comfortable to my colleagues. Sponsors generally want to solve a problem, not debate labels and language.”

Not Communicating the Level of Focus

A misalignment between the human factor(s) professionals and various stakeholders for level of focus on a project can impact the perceived value of the human factors work, often because the users think at a different/immediate level. As one participant explained: “There are pitfalls where you don’t want to focus on superficial, but legitimate human factors issues, like the colors are wrong or the lines should be thicker—that’s not making the work hard. And sometimes those aren’t even the issues we’re supposed to address. There are often tons of lower level human factors problems. And users will come to you and want to say, ‘Come here, I want to show you my system. It takes, five steps to get from here to here.’ Yes, that’s an issue, but that’s not the issue we’re being brought in to solve. It’s not at the right level. Someone should solve it, but. . .[not us].”

Equating Number of Events to Human Factors Value

Finally, one participant noted that one should not equate the number of related events to the value of a human factors evaluation or intervention: “I’ve been learning in the past years is to not to use the number of [patient safety] events to quantify the human factors value. I think like for each event they are unique and they have different failure modes. I think the uniqueness of human factors is that it is conducted in a systematic way. . . and the potential system failures, as that could contribute to the similar events. But the number of events that is reported in the event reporting system is not a reflection of how much the human factors evaluation contributes to that.”

Other Important Considerations

Timing

When attempting to demonstrate the value of a human factors project, a couple participants emphasized the importance of considering how to achieve this early during the planning or kick-off phase of the project. As one participant noted: “I think what we struggle with is the outcome measure. And so we’ve tried more and more to build that into our planning of how can we measure the outcome before we start the project.” Another participant explained: “. . .I do think I’m the voice at the kickoff meeting—‘What are we really trying to do here?’ ‘What do we care about?’ . . .when it comes to impact, you’ve got to get really focused. And you’re going to have to prioritize things, de-emphasize others, and so I want to start that conversation very early on. What’s the impact? What are we really trying to do here? . . .who do we want to influence here? Those are big questions and it helps if you can think about them very early on.”

Value From Who’s Perspective?

For processes and designs that impact both clinicians and patients, one participant described the importance of considering value from both perspectives. As one participant described: “I’m on a project where our hope is that this tool in the EHR will set the stage for better shared decision making between patients and clinicians about chronic pain treatment. And so we’re really thinking hard about how do you measure that? How do you know that you’ve improved that shared decision making? And from both the patient perspective and the clinician perspective? Because I think sometimes those two perspectives can be at odds with each other.”

The Patient as the Leverage Point

One strategy for demonstrating the value of human factors work is to make the work patient-focused when possible. One participant, who had previous experience working for the Department of Veterans Affairs (VA) explained: “I think you’re going to have a hard time leveraging around the system because the VA is an institution. And it’s a double, triple institution. It’s a military institution, healthcare institution, and it’s just an institution in an amongst itself. You’re not going to be able to change the structures, but you can have some impact on the processes. And I think the leverage point is the veteran. And the reason I think the veteran is the leverage point, is that I did not meet a soul there who was not dedicated to caring for the veterans.”

Discussion

Human factors in healthcare has advanced and matured substantially over the last couple of decades. For example, when the lead author began working for the Department of Veterans Affairs (VA) 20 years ago, only a handful of human factors professionals were based in the Veterans Health Administration (VHA), America’s largest integrated health care system, currently providing care at 1,321 health care facilities, including 172 medical centers and 1,138 outpatient sites of care of varying complexity (VHA outpatient clinics), serving 9 million enrolled Veterans each year. There was also a systemic lack of understanding of the human factors profession and how it could contribute to the design of healthcare systems and processes by enhancing patient safety, user experience, cognitive workload, etc. Today, human factors is on its way to being institutionalized within VA, in both the operations and research arms of the organization. Several informatics and clinical professionals undergo human factors training each year offered by the VA, including as part of the VA Health Informatics Certificate Program (VA AMIA 10x10: https://amia.org/education-events/amia-10x10-virtual-courses/amia-10x10-us-department-veterans-affairs). Despite this tremendous progress over the years, the progress does not seem to be linear, but rather more unpredictable, with turnover and a lack of sustained institutional memory for engaging with human factors professionals and leveraging human factors knowledge.

Based on years of experience in multiple healthcare organizations, participants in this project offered vital tips for building human factors capacity and demonstrating human factors value, including through the use of compelling storytelling skills, building relationships to achieve a network of internal champions, and adapting human factors language that will connect with the broader healthcare audience, among several other tips. Participants also warned against several pitfalls, such as failing to show humility in how we present our human factors work and how we can contribute. Additional considerations were offered, such as recognizing that healthcare as an institution can often be inflexible, and so finding the right leverage point (e.g., the patient) can be important for integrating human factors work. As Perry et al. (2021) explain, healthcare as a system and institution was not methodically designed, but rather “emerged,” making integration of human factors work much more challenging. Therefore, human factors professionals must take greater care finding the right leverage points and be willing to adapt (language, how human factors methods, and contributions are pitched, etc.) to achieve greater penetration in the healthcare institution.

When demonstrating the value of human factors work, being able to communicate a compelling story was identified an important skill by participants. Much of storytelling is a skill that develops naturally with years of experience in the human factors profession and engaging in a variety of projects. However, one can also take communications courses for developing persuasive storytelling skills. Many universities offer in person and online persuasive communication courses. There are also platforms like Coursera that offer relevant online communications courses.

Communicating the level of focus of a human factors project is crucial, as explained by some participants; a misunderstanding between the human factors professionals and clinical stakeholders can impact the perceived value of the human factors work. The lead author had a related experience 20 years ago when starting his first position with a VA patient safety center. After being assigned to assess a health information technology (IT) system for the intensive care unit (ICU), related to the interface, the main stakeholder, an ICU physician, made a comment along the lines of “I don’t want any recommendations about moving this button over here or anything like that.” Eventually, the lead author understood she wanted the focus on recommendations for the system design to make sure we weren’t accidently harming patients. There are human factors guidelines for interface design related to buttons, colors, etc. But she wanted an assessment of the larger level issues—the fundamental design, integration into clinical workflow, resilience of the system for patient safety type issues. Both levels of focus are legitimate, but if we as human factors professionals do not have the same alignment and understanding as the stakeholders for purpose/level of focus, it could impact the perceived value of the human factors work.

This project is primarily limited by a relatively small sample size of convenience. Gathering additional perspectives from other human factors professionals who are embedded in a wider range of healthcare organizations, or who engage with other healthcare institutions, may have yielded several more suggestions for demonstrating the value and impact of human factors work in healthcare.

Conclusion

We interviewed human factors healthcare professionals to understand their strategies for demonstrating the value and impact of their healthcare operations-based work. Analysis revealed several useful tips about what works well and pitfalls to avoid. This knowledge may help other human factors professionals embedded in healthcare institutions, or those who engage with healthcare institutions, achieve greater integration of human factors in healthcare. Although much progress has been made over the years for integrating human factors in healthcare processes and design, much more effort is needed toward institutionalizing human factors in healthcare.