Usability evaluation and assessment of a novel emergency department IT system developed using a cognitive systems engineering approach

Abstract

This paper presents the results of a usability evaluation conducted of an electronic Emergency Department information system (EDIS) prototype that was designed using a cognitive system engineering (CSE) approach. Participants were asked to complete tasks using the EDIS prototype, while thinking aloud about their interactions with the displays. Participants also completed subjective assessments of the system that related to 1) cognitive performance support objectives, 2) usability, usefulness, and frequency of system use, and 3) qualitative feedback. Mean scores were calculated for cognitive performance support objectives as well as usability, usefulness, and frequency of use for each display. Mean scores for all cognitive performance support objectives were six or higher. The mean usability score was five or higher, and the mean usefulness score was six or higher. Displays that provided information on individual patient status had the highest scores. Results from this evaluation regarding positive ratings for the cognitive support objectives provide evidence that CSE design methods can be used to understand the goals and objectives of medical work domains.

Get full access to this article

View all access options for this article.

References

Adams

J. A.

Humphrey

C. M.

Goodrich

M. A.

Cooper

J. L.

Morse

B. S.

Engh

Rasmussen

(2009). Cognitive Task Analysis for Developing Unmanned Aerial Vehicle Wilderness Search Support. Journal of Cognitive Engineering and Decision Making.

Bisantz

A. M.

(2008). Cognitive Engineering Applications in Health Care. Bridge, The, 38(4), 39–47.

Bisantz

A. M.

Burns

C. M.

(2008). Applications of Cognitive Work Analysis. Boca Raton, FL: CRC Press.

Bisantz

A. M.

Pennathur

P. R.

Fairbanks

R. J.

Perry

S. J.

Zwemer

Guarrera

T. K.

Wears

R. L.

(2010). Emergency department status boards: A case study in information systems transition. J Cogn Eng Decis Making, 4(1), 39–68.

Bisantz

A. M.

Roth

(2008). Analysis of cognitive work. The Annual Review of Human Factors and Ergonomics, 3, 1 – 43.

Bisantz

A. M.

Roth

E. M.

Brickman

Lin

McKinney

(2003). Integrating cognitive analyses in a large scale system design process. International Journal of Human-Computer Studies, 58, 177–206.

Burns

Enomoto

Momtahan

(2008). A cognitive work analysis of cardiac nurses performing teletriage. In Applications of Cognitive Work Analysis (2009th ed.). Boca Raton, FL: CRC Press.

Compton

D. W.

Ganjiang

Reid

P. P.

Grossman

J. H.

(2005). Building a Better Delivery System: A New Engineering/Health Care Partnership. Building (p. 276).

Fairbanks

R. J.

Caplan

Panzer

R. J.

(2005). Integrating Usability Into Development Of A Clinical Decision Support System. In HCI-International. Las Vegas, Nevada: Mira Digital Publishing .

10.

Gosbee

Lin

(2001). The Role of Human Factors Engineering in Medical Device and Medical System Errors. In Vincent

(Ed.), Clinical Risk Management: Enhancing Patient Safety (pp. 301–317; Chapter 16). London: BMJ Press.

11.

Guarrera

Stephens

Clark

McGeorge

Fairbanks

Perry

. . . Bisantz

A. M.

(2012). Engineering Better Health IT: Cognitive Systems Engineering of a Novel Emergency Department IT System. Proceedings of the International Symposium of Human Factors and Ergonomics in Healthcare, 1(1), 165.

12.

Healthcare Information Management Systems. (2009). Computerized Provider Order Entry. Vantage Point (Vol. 7).

13.

Koppel

Metlay

J. P.

Cohen

Abaluck

Localio

A. R.

Kimmel

S. E.

Strom

B. L.

(2005). Panacea or Pandora’s box: why computerized physician order entry systems facilitate as well as reduce medication errors. JAMA, 293, 1197–1203.

14.

Lin

Bisantz

A. M.

Fairbanks

Sui

Guarrera

Perry

S. J.

Brown

J. L.

(2008). Development of a Simulation Environment for the Study of IT Impact in the ED. 2008 Agency for Healthcare Research and Quality Annual Conference.

15.

Truxler

Roth

Scott

Smith

Wampler

(2012). Designing Collaborative Automated Planners for Agile Adaptation to Dynamic Change. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 56, 223–227.

16.

Vicente

K. J.

(1999). Cognitive Work Analysis. Mahwah, NJ: Lawrence Erlbaum Associates.

17.

Wears

R. L.

Perry

S. J.

Shapiro

M. J.

Beach

Croskerry

Behara

(2003). A Comparison of Manual and Electronic Status Boards in The Emergency Department: What’s Gained and What's Lost? In Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting (pp. 1415–1419). Denver, Colorado.

18.

Woods

Roth

E. M.

(1988). Cognitive Engineering: Problem Solving with tools. Human Factors, 30(4), 415–430.