High variability in Summative Usability Test methods & reporting among clinical informatics vendors complying with Federal certification requirements

Abstract

The Office of the National Coordinator for Health Information Technology (ONC) has been tasked with managing the integration of usability best practices into electronic health record development but, has provided little guidance on how to conduct Summative Usability Testing (SUT). We reviewed three aspects of SUT in four publicly available SUT vendor reports. Significant deviations from human factors engineering (HFE) and user-centered design (UCD) best practices and clear methodological errors were found in all four reports. The main sources of variation involved non-representative participant selection, instructions that led participant performance, and varied interpretation of results reporting requirements. Although ONC has made significant gains in implementing best practice UCD approaches, more explicit guidance may be needed to enhance integrity and reduce variation.