Drawing on human factors engineering to evaluate the effectiveness of health information technology

Introduction

There are substantial global investments being made in health information technology as governments and providers strive to improve the safety, quality and efficiency of healthcare. ¹ Accompanying this trend is a growing health information technology industry projected to be worth around $230 billion by 2020, with a range of new applications becoming available daily. ² These vary significantly in complexity, ranging from foundational technologies that impact on a range of care providers (e.g. electronic health records) to relatively discrete functionality affecting a limited range of settings and users (e.g. specialist apps for patients with chronic conditions) (see Box 1).

Box 1.

Examples of existing health information technologies and evaluation questions.

EHRs (also called electronic medical records (EMRs), electronic patient records (EPRs), depending on the focus): Issues including usability by non-clinical staff, changing relationships between clinician and patient, and new models of care.

Clinical decision support systems, computerised physician order entry, electronic medication administration records, patient barcode scanning, and related technologies for patient identification and medication management: Safety, interoperability and usability are all important for such technologies.

Technologies for medication administration and other therapies, including infusion pumps and syringe drivers (sometimes with dose error reduction software) for medication administration, and haemodialysis technology: Safety and usability are paramount across all settings, but particularly challenging for home use.

Monitoring technologies: Ease of use and interpretation of the information displayed are important.

In vitro diagnostics: As well as ease of use (particularly for delivering reliable results), it may be important to consider the care pathway to ensure people receive appropriate support depending on the diagnostic results.

Digital behaviour change interventions: Effective engagement is essential if the intervention is to be effective. Individual differences (knowledge, motivations, self-efficacy, etc.) influence engagement.

Randomised controlled trials of complex health information technology interventions are important and appropriate to evaluate effectiveness in many contexts, particularly if they are complemented by process and in-depth qualitative evaluations which yield insights into why the intervention was effective/ineffective and its likely generalisability. ³ But health information technology presents challenges for this traditional biomedical evaluation approach. It may have relatively small diffuse effects that are often difficult to trace and attribute as it involves redesigning existing organisational processes and ways of working (e.g. when implementing electronic health records). ⁴ The effect of the intervention can also be heavily shaped by the context and the way it is implemented ⁵ ; this may include, for example, varying organisational strategies, shifting responsibilities of healthcare professionals (e.g. towards a greater emphasis on data entry), increased managerial control (e.g. through review of data held within systems), changing power structures and whether or not computerised decision support is switched on. Health information technology is also often rapidly evolving and a certain user interface, for example, may no longer exist by the time a randomised controlled trial evaluative cycle (which typically takes years) has been completed. ⁶

New design and evaluation approaches are thus needed alongside randomised controlled trials to provide a broader array of evaluative approaches to investigate the effectiveness of different forms of health information technology. Building on our previous work highlighting the importance of continuous systemic health information technology evaluation, ⁷ we consider how evaluation approaches based on human factors engineering may help to address some of these needs.

In which contexts are randomised controlled trials appropriate?

Randomised controlled trials are of considerable importance to evaluating the effectiveness of interventions – including health information technology – because they have the potential to minimise the risk of bias, allocate confounders randomly between intervention arms and produce generalisable results. ⁸ This design is appropriate and indeed essential for some health information technology contexts where clear effects on a limited number of outcomes can be anticipated. For instance, if aiming to investigate the effectiveness of a safety/business-critical medical prototype device with discrete effects on health outcomes or if interventions are particularly expensive, then randomised controlled trial designs are likely to be an appropriate summative evaluative tool. Indeed, randomised controlled trials and alternatively quasi-randomised controlled trials, have been very useful in relation to evaluating health information technology, particularly when combined with embedded process and qualitative evaluations. ⁵

Why are randomised controlled trials problematic for some types of technology/context?

Our experiences have, however, indicated that there are a number of contexts where randomised controlled trials and quasi-randomised controlled trials are not feasible and/or appropriate. First, some health information technologies, such as electronic health records, are foundational and therefore have multiple small effects that are often hard to measure. Furthermore, with these types of technology, it can be difficult to establish appropriate controls, as health information technology implementations often involve large organisational transformations across care settings that are not directly comparable. Second, implementation context matters and should not be treated as a confounder. This may include the hospital environment into which systems are implemented, the leadership style and implementation strategy pursued, and user attributes such as personality and competencies. Third, many technologies rapidly evolve (e.g. by being refined, upgraded, customised) or they cease to exist altogether (e.g. apps), which means that the intervention may vary over time. As a result, many complex health information technology systems that are used across a range of contexts lack usability and fail to achieve their true potential as they are often used in ways other than intended. ⁹

To address some of these challenges, a human factors engineering approach may provide an alternative more agile approach to evaluating health information technology, as this takes into account the changing nature of technology while paying close attention to the context of system use and the variety of settings in which technologies may be implemented.

What is human factors engineering and when is it appropriate?

User-centred approaches to evaluation tend to be practically oriented, focusing on the iterative evaluation of the technology at hand; they are typically less concerned with producing generalisable results. ¹⁰ Although some traditional experimental evaluation approaches exist in these settings (e.g. performance and effectiveness analysis), there is an explicit focus on evaluating technology throughout the lifecycle, often involving iterative development and evaluation. ¹¹ An example of such an approach to system development is rapid application development. This employs iterative methods that allow technological prototypes to be developed quickly and then tested and refined in real-world settings, often based on user feedback. It is well suited for complex environments where effects of technologies and user requirements are hard to predict in advance (e.g. when technologies require a high degree of interactivity). The approach may draw on cognitive task analysis, where user needs and workflows are assessed before new technology is introduced, followed by detailed analysis surrounding how new systems affect existing practices. ¹² Action research can be a useful methodology in human factors engineering; however, human factors engineering employs a variety of research methods and has an explicit focus on design, particularly of technology.

Human factors engineering is thus a usability engineering-based approach to ‘designing systems for human use’, helping to develop systems over time to bring maximum benefits to users. ¹² It includes prospective evaluation approaches that allow systems to be refined in ways that promote their effective use over time. The underlying assumption is that health information technology only works if it is usable and fits with users’ practices. Human factors engineering helps to design usable and useful systems. ¹³ Methods focus on obtaining a clear understanding of what task needs to be undertaken (the aim) and helping to design systems that assist and motivate users to accomplish this task (the tool). From this perspective, evaluation of technology should therefore focus on exploring its suitability for accomplishing a task from the perspective of users, the degree to which technologies support and enhance human effectiveness, and the degree to which they fit within their context of use.

Although attention to human factors engineering in design and development of systems is important, there is also growing recognition that ongoing design iterations are an inherent feature of the technology lifecycle. ¹⁴ These iterations may involve co-evolution of systems and users, where technologies and ways of using them are changed over time to suit contexts of use. They are often difficult to anticipate ahead of time and can create opportunities for system improvement as well as expose design limitations that may be important to address to improve workflows and mitigate potential safety risks. ¹⁵ Steps involved in this prospective evaluation process include understanding current practices and needs, identifying possible design solutions, checking proposed solutions against needs, testing of implemented systems with users, and continuous co-evolution between design and use (see Table 1). ¹⁶ Techniques and tools used in human factors engineering are outlined in Table 2; see also Cassano-Piche et al. ¹⁷ and Preece et al. ¹⁸

Table 1.

Lifecycle perspective of implementing health information technology and human factors engineering-informed lines of inquiry.

Lifecycle stage	HFE questions
Conceptualisation	What are the organisation’s or individual’s needs?
Project initiation	What kinds of solutions (not necessarily technological) are there to address those needs?
Functional specification	If the solution is technological, then what functionalities are likely to address the specified needs? Who will the users of the technology be?
Drafting a business case	(For each group of users): how will the system fit in their workflow? How will their workflow need to be adapted to exploit the new technology? What are the costs and benefits to them (in their role) in using the new system, and how the benefits outweigh the costs?
Procurement/tendering	What technologies exist and which ones are likely to address the organisational/user needs? Does a new technology need to be designed and implemented?
System choice	How does the new technology support the long-term agenda of the organisation (e.g. for patient engagement/shared care, interoperating with other new systems)? How easy is the new system to use? How safe is it (e.g. limiting effects of human error)? (Particularly important if it is only used occasionally by a particular user group)
Contracting	How can organisations and suppliers work together to implement a technology that is usable and brings benefits to users?
Pre-implementation	How exactly do workflows need to be changed across different professions and settings?
Implementation	How have planned changes to workflows played out in reality? Are there adverse consequences for some groups of users?
System optimisation	How can users and organisations help to shape systems to better fit with their contexts of use? What are the opportunities for improvement and what are design limitations?

Table 2.

Methodologies commonly used in human factors engineering.

Tool	Features	Useful for	Considerations
Observation (sometimes called ‘ethnography’)	Observing people working and using health information technology	Understanding what people do in practice and what they need	Usefully complemented by interviews or contextual inquiry, to better understand what is observed
Interviews (usually semi-structured)	Interviewing people about their work, their experiences of health information technology, their requirements for future health information technology, etc.	Understanding people’s perceptions and experiences	People have difficulty reporting accurately on what they do
Contextual inquiry	Combining observations and interviews to understand work and the use of health information technology	Gaining insights for design based on a better understanding of people’s work and activities	Takes place within the context where people use health information technology
Diary studies	Participants maintain a diary of relevant thoughts and experiences when using the health information technology	Gathering information about situated use (e.g. of mobile health information technology)	Quality of data is dependent on commitment of participants
Task analysis	Systematically decomposing tasks (that the health information technology supports) into sub-tasks to analyse the sequence and performance criteria	Supports systematic thinking about user tasks and how they are achieved with the health information technology	Should be based on empirical data of real user tasks
Cognitive task analysis	Systematic analysis of the cognitive tasks (user’s thought processes) when using the health information technology	Analysis of issues such as likely errors, mental workload and compatibility between tasks as defined by health information technology and the way people think about their tasks	Requires expertise in cognitive science
Rapid application development	Employs iterative methods that allow technological prototypes to be developed quickly and then tested and refined in real-world settings, often based on user feedback	Well suited for complex environments where the effects of technologies and user requirements are hard to predict in advance (e.g. when technologies require a high degree of interactivity)	Close collaboration between users, implementers and developers is essential
Heuristic evaluation	A checklist approach to check the device interface for usability and safety based on ‘rules of thumb’	Checking for obvious problems at early stages of development	Needs expertise in understanding and interpreting the heuristics. Dependent on the expertise (and biases) of the evaluators
Cognitive walkthrough	An expert review approach that involves ‘walking through’ the steps of an interaction between user and device, reasoning about possible user errors. One common form of cognitive task analysis	Early review, focusing on user cognition, requiring access to a prototype health information technology	Should be conducted by experts in cognitive science. Assumes that the health information technology is ‘walk up and use’
User testing (often with think-aloud)	Testing the health information technology with representative users in a simulated use environment. Users are often invited to articulate thoughts while interacting with health information technology	Identifying which health information technology features people find easy to use and which cause problems	Participants should be representative of the intended user population(s) and tasks used in should provide good coverage of real-world use. Yields insights into health information technology design but not how it fits in the broader work context
Randomised controlled trial	Testing the health information technology with representative users under controlled conditions to test a hypothesis about health outcomes	Testing the effectiveness of health information technology for achieving health outcomes	Need a valid comparator and conditions need to be as realistic as possible while remaining controlled

Towards re-conceptualising health information technology evaluation approaches

Both randomised controlled trial-based and human factors engineering-based approaches to evaluation are essential for evaluating the effectiveness of health information technology, but it is important to recognise that the appropriateness of methods depends on the needs emerging from different contexts/technologies. The choice of methods should be determined by the type of health information technology to be evaluated, its stage of development and the key evaluation questions in that situation. For example, human factors engineering approaches work well for technologies that have distributed effects and can be tested and refined in collaboration with users in real-world settings. In comparison, randomised controlled trials are necessary for pre-market testing of safety-critical medical devices and for determining the impact of technologies on health outcomes. ¹⁹ We propose a decision tree to guide evaluators in relation to choice of methods in Figure 1.

Figure 1.

Flow diagram guiding the choice of evaluation approaches.

A key future activity should include establishing consensus among evaluators on which approaches are best suited to different types of technologies and evaluation questions, and establishing key influence diagrams to consider appropriate surrogate markers. ²⁰ For instance, safety/business-critical technologies (e.g. infusion devices, haemodialysis machines, clinical decision support systems) are likely to require extensive upfront development (ideally in close collaboration with users) and extensive effectiveness testing, while prototypes of less critical devices (e.g. activity trackers) may be tested and refined in close collaboration with users in real-world settings without significant safety implications.

Conclusions

Traditional randomised controlled trial-based evaluation paradigms, although suitable for determining effectiveness surrounding health outcomes, are not appropriate for evaluating the effectiveness of different types of health information technology at different stages of development. Human factors engineering-based evaluation approaches can help to address these shortcomings, particularly in relation to designing systems that fulfil user needs and involve users throughout the development process.