Health Information Technology (HIT) systems have become integral to today's health care. However, the success of such systems depends much on their usability because poor usability invites more errors and cognitive loads, ultimately affecting healthcare workers’ efficiency and well-being.
Objectives
To identify and summarize the most applied methods of HIT system evaluation, focusing on their impact on work environments and ergonomic practices.
Methods
A scoping review of the literature available from January 2000 up to April 2024 was carried out. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews checklist (PRISMA-ScR) was used. Searches were conducted in four databases. Literature was considered for inclusion if it focused on the HIT, provided information related to the usability of these systems, and provided empirical results of the usability testing. It was written in English.
Results
A total of 591 articles were initially retrieved. Manual searches of reference lists added 21 more articles. Ultimately, 48 articles met the inclusion criteria. The most used method for usability testing was the self-response category.
Conclusions
While qualitative and quantitative methods have been used for usability testing, physiological methods remain underutilized. Eye tracking appeared in only a few studies, and its relationship to usability is still unclear. This study highlights the need to integrate ergonomic principles into HIT system design to enhance usability and support healthcare professional working conditions.
ChaudhryBWangJWuS, et al.Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med2006; 144: 742–752.
2.
BuntinMBBurkeMFHoaglinMC,et al.The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health Aff2011; 30: 464–471.
3.
ZahabiMKaberDBSwangnetrM. Usability and safety in electronic medical records interface design: a review of recent literature and guideline formulation. Hum Factors2015; 57: 805–834.
4.
ClarkeMASteegeLMMooreJL, et al. (eds). Addressing human computer interaction issues of electronic health record in clinical encounters. In: Design, user experience, and usability health, learning, playing, cultural, and cross-cultural user experience: second international conference, DUXU 2013, held as part of HCI international 2013, Las Vegas, NV, USA, 21–26 July 2013, Proceedings, Part II 2. Springer, 2013.
5.
ZhengKHaftelHMHirschlRB, et al.Quantifying the impact of health IT implementations on clinical workflow: a new methodological perspective. J Am Med Inf Assoc2010; 17: 454–461.
6.
RatwaniRMReiderJSinghH. A decade of health information technology usability challenges and the path forward. Jama2019; 321: 743–744.
7.
KhairatSColemanCOttmarP, et al.Physicians’ gender and their use of electronic health records: findings from a mixed-methods usability study. J Am Med Inf Assoc2019; 26: 1505–1514.
8.
KhairatSColemanCNewlinT, et al.A mixed-methods evaluation framework for electronic health records usability studies. J Biomed Inf2019; 94: 103175.
9.
RatwaniRMSavageEWillA, et al.A usability and safety analysis of electronic health records: a multi-center study. J Am Med Inf Assoc2018; 25: 1197–1201.
10.
RizviRFMarquardJLSeywerdMA, et al. Usability evaluation of an EHR's clinical notes interface from the perspective of attending and resident physicians: an exploratory study. MEDINFO 2017: precision healthcare through informatics. Amsterdam: IOS Press, 2017, pp.1128–1132.
11.
GoldJAStephensonLEGorsuchA, et al.Feasibility of utilizing a commercial eye tracker to assess electronic health record use during patient simulation. Health Inf J2016; 22: 744–757.
12.
WaljiMFKalenderianETranD, et al.Detection and characterization of usability problems in structured data entry interfaces in dentistry. Int J Med Inf2013; 82: 128–138.
13.
WrightMCDunbarSMacphersonBC, et al.Toward designing information display to support critical care. Appl Clin Inf2016; 7: 912–929.
14.
XuLWenDZhangX,et al.Assessing and comparing the usability of Chinese EHRs used in two Peking University hospitals to EHRs used in the US: a method of RUA. Int J Med Inf2016; 89: 32–42.
15.
AhmedAChandraSHerasevichV, et al.The effect of two different electronic health record user interfaces on intensive care provider task load, errors of cognition, and performance. Crit Care Med2011; 39: 1626–1634.
16.
ChoHKeenanGMadandolaOO, et al.Assessing the usability of a clinical decision support system: heuristic evaluation. JMIR Hum Factors2022; 9: e31758.
17.
NguyenKAPatelHHaggstromDA, et al.Utilizing a user-centered approach to develop and assess pharmacogenomic clinical decision support for thiopurine methyltransferase. BMC Med Inf Decis Making2019; 19: 1–13.
18.
KilsdonkEPeuteLRiezebosRJ, et al.Uncovering healthcare practitioners’ information processing using the think-aloud method: from paper-based guideline to clinical decision support system. Int J Med Inf2016; 86: 10–19.
19.
CaseySDReedMELeMasterC, et al.Physicians’ perceptions of clinical decision support to treat patients with heart failure in the ED. JAMA Netw Open2023; 6: e2344393–e.
20.
PruittZHoweJLKrevatSA, et al.Development and pilot evaluation of an electronic health record usability and safety self-assessment tool. JAMIA Open2022; 5: ooac070.
21.
KhajoueiRde JonghDJaspersMW. Usability evaluation of a computerized physician order entry for medication ordering. In: Medical informatics in a united and healthy. Europe: IOS Press, 2009, p.532–536.
22.
KhajoueiRPeekNWierengaP, et al.Effect of predefined order sets and usability problems on efficiency of computerized medication ordering. Int J Med Inf2010; 79: 690–698.
23.
Hayavi-HaghighiMHDavoodiSTeshniziSH,et al.Usability evaluation of electronic prescribing systems from physicians’ perspective: a case study from southern Iran. Inf Med Unlocked2024; 45: 101460.
24.
AufeggerLSerouNChenS,et al.Evaluating users’ experiences of electronic prescribing systems in relation to patient safety: a mixed methods study. BMC Med Inf Decis Making2020; 20: –8.
25.
KauppinenHAhonenRMäntyselkäP,et al.Medication safety and the usability of electronic prescribing as perceived by physicians—A semistructured interview among primary health care physicians in Finland. J Eval Clin Pract2017; 23: 1187–1194.
26.
RogersMLSockolowPSBowlesKH, et al.Use of a human factors approach to uncover informatics needs of nurses in documentation of care. Int J Med Inf2013; 82: 1068–1074.
27.
KhajoueiRAbbasiR. Evaluating nurses’ satisfaction with two nursing information systems. CIN Comput Inf Nurs2017; 35: 307–314.
28.
FarzandipourMNabovatiETadayonH,et al.Usability evaluation of a nursing information system by applying cognitive walkthrough method. Int J Med Inf2021; 152: 104459.
29.
AtashiAKhajoueiRAziziA,et al.User interface problems of a nationwide inpatient information system: a heuristic evaluation. Appl Clin Inf2016; 7: 89–100.
30.
FarzandipourMNabovatiEZaeimiG-H,et al.Usability evaluation of three admission and medical records subsystems integrated into nationwide hospital information systems: heuristic evaluation. Acta Inf Med2018; 26: 33.
31.
FarrahiRRangraz JeddiFNabovatiE, et al.The relationship between user interface problems of an admission, discharge and transfer module and usability features: a usability testing method. BMC Med Inf Dec Making2019; 19: 1–8.
32.
FarzandipourMJabaliMSNickfarjamAM,et al.Usability evaluation of selected picture archiving and communication systems at the national level: analysis of users’ viewpoints. Int J Med Inf2021; 147: 104372.
33.
EsfahaniMZKhajoueiRBaneshiMR. Augmentation of the think-aloud method with users’ perspectives for the selection of a picture archiving and communication system. J Biomed Inf2018; 80: 43–51.
34.
RussALZillichAJMeltonBL, et al.Applying human factors principles to alert design increases efficiency and reduces prescribing errors in a scenario-based simulation. J Am Med Inf Assoc2014; 21: e287–ee96.
35.
XieMWeingerMGreggW,et al.Presenting multiple drug alerts in an ambulatory electronic prescribing system. Appl Clin Inf2014; 5: 334–348.
36.
JeddiFRNabovatiEBighamR,et al.Usability evaluation of a comprehensive national health information system: a heuristic evaluation. Inf Med Unlocked2020; 19: 100332.
37.
JeddiFRNabovatiEBighamR,et al.Usability evaluation of a comprehensive national health information system: relationship of quality components to users’ characteristics. Int J Med Inf2020; 133: 104026.
38.
BrownBBalatsoukasPWilliamsR, et al.Multi-method laboratory user evaluation of an actionable clinical performance information system: implications for usability and patient safety. J Biomed Inf2018; 77: 62–80.
39.
BrownBBalatsoukasPWilliamsR, et al.Interface design recommendations for computerised clinical audit and feedback: hybrid usability evidence from a research-led system. Int J Med Inf2016; 94: 191–206.
40.
MohammadzadehZNickfarjamAMAtoofF, et al.Evaluating usability of computerized physician order entry systems: insights from a developing nation. Inf Med Unlocked2024; 47: 101487.
41.
PoncetteA-SMoschLKStabloL, et al.A remote patient-monitoring system for intensive care medicine: mixed methods Human-Centered design and usability evaluation. JMIR Hum Factors2022; 9: e30655.
42.
PoonSJGreenwood-EricksenMBGishRE, et al.Usability of the Massachusetts prescription drug monitoring program in the emergency department: a mixed-methods study. Acad Emerg Med2016; 23: 406–414.
43.
Rezaei-HachesuPPesianianEMohammadianM. Evaluating usability of radiology information systems in hospitals of Tabriz University of medical sciences. Acta Inf Med2016; 24: 42.
44.
ScandurraIHägglundMMoströmD,et al.Heuristic evaluation extended by user analysis: a fast and efficient method to identify potential usability problems in health information systems. J Inf Technol Healthcare2006; 4: 317–325.
45.
ShieldsCCunninghamSGWakeDJ, et al.User-centered design of a novel risk prediction behavior change tool augmented with an artificial intelligence engine (MyDiabetesIQ): a sociotechnical systems approach. JMIR Hum Factors2022; 9: e29973.
46.
AgharezaeiZKhajoueiRAhmadiaL,et al.Compliance with design principles: a case study of a widely used laboratory information system. East Mediterr Health J2020; 26: 1456–1464.
47.
AlinejhadANouraniAJebraeilyM. Usability evaluation of the user interface in electronic prescribing systems of Iran health insurance organization and social security organization. J Health Adm2022; 25: 78–94.
48.
BlakeneyEA-RDardasTZierlerBK,et al.Development and usability testing of a system to detect adverse events and medical mistakes. CIN Comput Inf Nurs2023; 41: 330–337.
49.
DodsonCLaymanL. Refinement of a pharmacogenomics app for dosing guidelines for oncology: findings from the usability evaluation. Ann Transl Med2022; 10: 1261–1261.
50.
van Engen-VerheulMMPeuteLWde KeizerNF, et al.Optimizing the user interface of a data entry module for an electronic patient record for cardiac rehabilitation: a mixed method usability approach. Int J oMed Inf2016; 87: 15–26.
51.
FraccaroPVigoMBalatsoukasP, et al.Presentation of laboratory test results in patient portals: influence of interface design on risk interpretation and visual search behaviour. BMC Med Inf Decis Making2018; 18: 1–12.
52.
SavoyAMilitelloLGPatelH, et al.A cognitive systems engineering design approach to improve the usability of electronic order forms for medical consultation. J Biomed Inf2018; 85: 138–148.
53.
SauroJLewisJR. Quantifying the user experience: practical statistics for user research. United States: Morgan Kaufmann, 2016.
54.
LewisC. Using the” thinking-aloud” method in cognitive interface design. Yorktown Heights, NY: IBM TJ Watson Research Center, 1982.
55.
NielsenJ. Response times: the three important limits. United States: Usability Engineering, 1993.
56.
SimonHAEricssonKA. Protocol analysis: Verbal reports as data. (No Title). 1984.
57.
NielsenJ. Usability engineering. United States: Morgan Kaufmann Publishers In. Inc–1993, 1993.
58.
SalzTSchnallRBMcCabeMS, et al.Incorporating multiple perspectives into the development of an electronic survivorship platform for head and neck cancer. JCO Clin Cancer Inf2018; 2: 1–15.
59.
BowmanS. Impact of electronic health record systems on information integrity: quality and safety implications. Perspect Health Inf Manage2013; 10: 1–19.
60.
ConfortiDCostanzoDLaganiV, et al.HEARTFAID: a knowledge based platform for supporting the clinical management of elderly patients with heart failure. J Inf Technol Healthcare2006; 11: 46–51.
61.
JaspersMW. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inf2009; 78: 340–353.
SchaarupCHartvigsenGLarsenLB, et al. Assessing the potential use of eye-tracking triangulation for evaluating the usability of an online diabetes exercise system. MEDINFO 2015: eHealth-enabled health. Amsterdam: IOS Press, 2015, pp.84–88.
64.
CardSKMoranTPNewellA. The keystroke-level model for user performance time with interactive systems. Commun ACM1980; 23: 396–410.
65.
JohnBE. Information processing and skilled behavior. HCI models, theories, and frameworks. 2003.
66.
AlfimtsevABasarabMDevyatkovV,et al.A new methodology of usability testing on the base of the analysis of user’s electroencephalogram. J Comput Sci Appl2015; 3: 105–111.
67.
LauferLNémethB. Your skin knows when you will jump. In: Proceedings of the 14th European conference on cognitive ergonomics: invent! explore!. London, United Kingdom: Association for Computing Machinery, 2007, p.277–280.
68.
ZakiTIslamMN. Neurological and physiological measures to evaluate the usability and user-experience (UX) of information systems: a systematic literature review. Comput Sci Rev2021; 40: 100375.
69.
TaffeseT.A review of using EEG and EMG psychophysiological measurements in user experience research. 2017.
70.
GanglbauerESchrammelJDeutschS, et al. (eds). Applying psychophysiological methods for measuring user experience: possibilities, challenges and feasibility. In: Workshop on user experience evaluation methods in product development. New Jersey: Citeseer, 2009, pp.1–4.
71.
KivikangasJMChanelGCowleyB, et al.A review of the use of psychophysiological methods in game research. J Gaming Virtual Worlds2011; 3: 181–199.
