Formative Usability Evaluation of a Novel Tool for Medication Reconciliation

Abstract

To decrease medication errors, a common cause of injury to patients, we developed a novel electronic tool to facilitate asynchronous communication between healthcare professionals (HCPs) and patients for medication reconciliation. However, it was unknown whether the tool adequately supported HCPs’ usability needs. Our objective was to conduct an iterative usability evaluation of the tool with physicians, nurses, and pharmacists, in preparation for a randomized controlled trial. We hypothesized that we would identify design weaknesses that could be addressed via interface modifications prior to the trial. We completed a mixed-method, formative usability evaluation with 20 HCPs in the Veterans Affairs (VA) Health Services Research and Development, Human-Computer Interaction and Simulation Laboratory located within a major medical center. The tool in this study is formally known as the Secure Messaging for Medication Reconciliation Tool (SMMRT). The evaluation consisted of four sequential steps: 1) phase I usability testing to assess the baseline tool along with small, iterative design changes throughout testing; 2) heuristic evaluation; 3) implement major design changes that incorporate findings from previous steps; and 4) phase II usability testing to assess the implemented design changes and further refine the tool. This presentation focuses on steps 1 and 4 related to usability testing.

During testing, HCPs worked through a real case consisting of a patient discharged from the hospital within the past 30 days who had at least 5 outpatient medications. We collected data on efficiency, usability errors, and participants’ satisfaction, along with participants’ ability to detect and address three distinct types of medication errors via the tool. For the latter, we inserted three safety probes into the simulation: 1) a missing medication (i.e., omission); 2) an extraneous medication (i.e., commission); and 3) an inaccurate dose (i.e., dose discrepancy). Data were analyzed descriptively, rather than via statistical comparisons, due to the formative and iterative nature of this research.

There was no indication of efficiency gains during iterative prototyping and testing. Highlights of usability errors included confusion about medication entry fields; incorrect assumptions regarding medication list accuracy; inadequate medication information sorting and organization; and premature closure. Additionally, HCPs described usability errors that might occur in clinical practice. For example, medication images on the tool may not match what is dispensed to patients. HCPs also expressed concern that medication updates made via the tool may not be consistently updated in the electronic health record. In terms of satisfaction, HCPs’ ratings tended to increase as design modifications were implemented. After phase II usability testing, their overall satisfaction was favorable. Finally, for each of the three safety probes, 50% or fewer of HCPs identified the associated medication error.

This research illustrates the importance of usability evaluations as a precursor to randomized trials of health information technology. Our multi-step approach to usability testing, with heuristic evaluation at the midpoint, may inform the design of other usability evaluations. While efficiency gains were not realized, user satisfaction improved. The inclusion of safety probes was especially valuable, since probes allowed us to assess error detection rates. There may be opportunities for human factors professionals to expand the sophistication and types of probes used in future healthcare research. Future studies are needed to develop more advanced design approaches that facilitate healthcare professionals’ detection of medication errors.