Scenario Development for Decision Support System Evaluation

Abstract

This paper introduces a methodology for developing scenarios representative of the cognitive and collaborative challenges inherent in a domain of practice for evaluating Decision Support Systems (DSS). Explicit links are made between particular aspects of the DSS and specific cognitive and collaborative demands they are intended to support. The effectiveness of the DSS in supporting performance can then be systematically probed by creating scenarios that are informed by an understanding of individual and team cognitive processing factors, fundamental relationships within the domain, and known complicating factors that can arise in the domain to challenge cognitive and collaborative performance. This paper introduces a set of explicit artifacts to systematically create such scenarios to provide feedback on the viability of the DSS design concepts (e.g., are the hypothesized positive impacts of the DSS realized?), as well as feedback on additional unanticipated requirements for support.

Get full access to this article

View all access options for this article.

References

Carrera

J. P.

Easter

J. R.

Roth

E. M.

(1996). Simulator testing of the Westinghouse AWARE alarm management system. In Proceedings of the IAEA Specialists meeting on Experience and Improvements in Advanced Alarm Annunciation Systems in NPPs, Sept. 16–20, 1996.

Elm

W.C.

Potter

S.S.

Gualtieri

J.W.

Roth

E.M.

Easter

J.R.

(in press). Applied Cognitive Work Analysis: A pragmatic Methodology for Designing Revolutionary Cognitive Affordances. In Hollnagel

(Ed) Handbook for Cognitive Task Design. London: Lawrence Erlbaum Associates, Inc.

Mumaw

R. J.

Roth

E. M.

How to be more devious with a training simulator: Redefining scenarios to emphasize cognitively difficult situations. 1992 Simulation MultiConference: Nuclear Power Plant Simulation and Simulators, Orlando, FL, April 6–9, 1992.

Oser

Gualtieri

Cannon-Bowers

Salas

(1999). Training team problem solving skills: An event based approach. Computers in Human Behavior, 15,441–462.

Patterson

E. S.

Roth

E. M.

Woods

D. D.

(2001). Predicting vulnerabilities in computer-supported inferential analysis under data overload. Cognition, Technology and Work., 3, 224–237.

Roth

E. M.

Mumaw

R. J.

Lewis

P. M.

(1994). An Empirical Investigation of Operator Performance in Cognitively Demanding Simulated Emergencies. Washington D. C.: U. S. Nuclear Regulatory Commission (NUREG/CR-6208).

Roth

E. M.

Lin

Kerch

Kenney

S. J.

Sugibayashi

(2001) Designing a first-of-a kind group view display for team decision making: a case study. In Salas

Klein

(Eds.) Linking Expertise and Naturalistic Decision Making (pp. 113–135). Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc.

Vicente

K. J.

(1999). Cognitive work analysis: Towards safe, productive, and healthy computer-based work. Mahwah, NJ: Erlbaum.

Woods

D.D.

Patterson

E.S.

(in press). How Unexpected Events Produce An Escalation Of Cognitive And Coordinative Demands. Hancock

P.A.

Desmond

P.A.

(Eds.). Stress Workload and Fatigue. Lawrence Erlbaum, Hillsdale NJ.

10.

Woods

D.D.

Johannesen

Cook

Sarter

(1994). Behind Human Error: Cognitive Systems, Computers and Hindsight. Crew Systems Ergonomic Information and Analysis Center, WPAFB, Dayton OH.

11.

Woods

D.D.

Sarter

(1993). Evaluating the Impact of New Technology on Human-Machine Cooperation. In Wise

Hopkin

V. D.

Stager

(Eds.) Verification and Validation of Complex Systems: Human Factors Issues, Springer-Verlag, Berlin.