An Empirical Comparison of Methods for Eliciting and Modeling Expert Knowledge

Abstract

The goal of this project in Human-Centered Computing was to apply a variety of methods of Cognitive Task Analysis (CTA) and Cognitive Field Research (CFR) to support a complete process going all the way from knowledge elicitation to leverage point identification and then to system prototyping, and also use this as an opportunity to empirically compare and evaluate the methods. The research relied upon the participation of expert, journeyman, and apprentice weather forecasters at the Naval Training Meteorology and Oceanography Facility at Pensacola Naval Air Station. Methods included Protocol Analysis, a number of types of structured interviews, workspace and work patterns analysis, the Critical Decision Method, the Knowledge Audit, Concept Mapping, and the Cognitive Modeling Procedure. The methods were compared in terms of (1) their yield of information that was useful in modeling expert knowledge, (2) their yield in terms of identification of leverage points (where the application of new technology might bring about positive change), and (3) their efficiency. Efficiency was gauged in terms of total effort (time to prepare to run a procedure, plus time to run the procedure, plus time to analyze the data) relative to the yield (number of leverage points identified, number of propositions suitable for use in a model of domain knowledge). CTA/CFR methods supported the identification of dozens of leverage points and also yielded behaviorally- validated models of the reasoning of expert forecasters. Knowledge modeling using Concept-Mapping resulted in over a thousand propositions covering domain knowledge. The Critical Decision Method yielded a number of richly-populated case studies with associated Decision Requirements Tables. Results speak to the relative efficiency of various methods of CTA/CFR, and also the strengths of each of the methods. In addition to extending our empirical base on the comparison of knowledge elicitation methods, a deliverable from the project was a knowledge model that illustrates human-centered computing in that it integrates training support and performance aiding.

Get full access to this article

View all access options for this article.

References

Ausubel

D. P.

Novak

J. D.

Hanesian

(1978). Educational psychology: A cognitive view (2nd ed.). New York: Holt, Rinehart and Winston.

Buckley

B. W.

Leslie

L. M.

(2000). The Australian Boxing Day storm of 1998—Synoptic description and numerical simulations. Weather & Forecasting, 16, 543–558.

Burton

A. M.

Shadbolt

N. R.

Hedgecock

A. P.

Rugg

(1987). A formal evaluation of a knowledge elicitation techniques for expert systems: Domain 1. In Moralee

D. S.

(Ed.), Research and development in expert systems, Vol 4. (pp. 35–46). Cambridge: University Press.

Chi

M. T. H.

Glaser

Farr

M. J.

(Eds.) (1988). The nature of expertise. Mahwah, NJ: Erlbaum.

Cooke

N. M.

(1994). Varieties of knowledge elicitation techniques. International Journal of Human-Computer Studies, 41, 801–849.

Duda

R. O.

Shortliffe

E. H.

(1983). Expert systems research. Science, 220, 261–268.

Evans

A. W.

Jentsch

Hitt

J. M.

Bowers

Salas

(2001). Mental model assessments: Is there convergence among different methods? In Proceedings of the Human Factors and Ergonomics Society 45th Annual Meeting, (pp. 293–296). Santa Monica, CA: Human Factors and Ergonomics Society.

Hoffman

R. R.

(1987, Summer). The problem of extracting the knowledge of experts from the perspective of experimental psychology. The AI Magazine, 8, 53–67.

Hoffman

R. R.

(Ed.). (1992). The psychology of expertise: Cognitive research and empirical AI. New York: Springer Verlag.

10.

Hoffman

R. R.

Shadbolt

Burton

A. M.

Klein

G. A.

(1995). Eliciting knowledge from experts: A methodological analysis. Organizational Behavior and Human Decision Processes, 62, 129–158.

11.

Hoffman

R. R.

Coffey

J. W.

Carnot

M. J.

(2000, November). Is there a “fast track” into the black box?: The Cognitive Models Procedure. Poster presented at the 41st annual meeting of the Psychonomics Society, New Orleans, LA.

12.

Hoffman

R. R.

Crandall

Shadbolt

(1998). A case study in cognitive task analysis methodology: The Critical Decision Method for the elicitation of expert knowledge. Human Factors, 40, 254–276.

13.

Hoffman

R. R.

Shadbolt

Burton

A. M.

Klein

G. A.

(1995). Eliciting knowledge from experts: A methodological analysis. Organizational Behavior and Human Decision Processes, 62, 129–158.

14.

Hoffman

R. R.

Woods

D. D.

(2000). Studying cognitive systems in context. Human Factors, 42, 1–7.

15.

Hollnagel

Woods

D. D.

(1983). Cognitive Systems Engineering: New wine in new bottles. International Journal of Man-Machine Studies, 18, 583–600.

16.

Militello

L. G.

Hutton

R. J. B.

(1998). Applied Cognitive Task Analysis (ACTA): A practitioner's toolkit for understanding cognitive task demands. Ergonomics, 41, 1618–1641. Special Issue: Task Analysis.

17.

Novak

J. D.

(1998). Learning, creating, and using knowledge. Mahwah, NJ: Erlbaum.

18.

Potter

S. S.

Roth

E. M.

Woods

D. D.

Elm

W. C.

(2000). Bootstrapping multiple converging cognitive task analysis techniques for system design. In Schraagen

J. M.

Chipman

S. F.

Shalin

V. L.

(Eds.), Cognitive task analysis (pp. 317–340). Mahwah, NJ: Erlbaum.

19.

Rasmussen

(1992). Use of field studies for design of workstations for integrated manufacturing systems. In Helander

Nagamachi

(Eds.), Design for manufacturability: A systems approach to concurrent engineering and ergonomics (pp. 317–338). London: Taylor and Francis.

20.

Shanteau

(1992). Competence in experts: The role of task characteristics. Organizational Behavior and Human Decision Processes, 53, 252–266.

21.

Thorsden

M. L.

(1991). A comparison of two tools for cognitive task analysis: Concept Mapping and the Critical Decision Method. In Procedings of the Human Factors Society 35th Annual Meeting (pp. 283–285). Santa Monica: CA: Human Factors Society.

22.

Vicente

(1999). Cognitive work analysis: Toward safe, productive, and healthy computer-based work. Mahwah, NJ: Erlbaum.