The Critical Decision Method (CDM) is an approach to cognitive task analysis. The method involves multiple-pass event retrospection guided by probe questions. The CDM has been used in the elicitation of expert knowledge in diverse domains and for applications including system development and instructional design. The CDM research illustrates the sorts of knowledge representation products that can arise from cognitive task analysis (e.g., Situation Assessment Records, time lines, decision requirements). The research also shows how one can approach methodological issues surrounding cognitive task analysis, including questions about data quality and method reliability, efficiency, and utility. As cognitive task analysis is used more widely in the elicitation, preservation, and dissemination of expert knowledge and is used more widely as the basis for the design of complex cognitive systems, and as projects move into even more field applications and real-world settings, the issues we discuss become increasingly critical.
Get full access to this article
View all access options for this article.
References
1.
Adelman, L. (1989). Management issues in knowledge elicitation. IEEE Transactions on Systems, Man and Cybernetics, 19, 483–488.
2.
Bainbridge, L. (1979). Verbal reports as evidence of the process operator's knowledge. International Journal of Man-Machine Studies, 11, 411–436.
3.
Beach, L. R., & Lipshitz, R. (1993). Why classical decision theory is an inappropriate standard for evaluating and aiding most human decision making. In G. A. Klein, J. Orasanu, R. Calderwood, & C. E. Zsambok (Eds.), Decision making in action: Models and methods (pp. 21–35). Norwood, NJ: Ablex.
4.
Benjafield, J. (1969). Evidence that “thinking aloud” constitutes an externalization of inner speech. Psychonomic Science, 15, 83–84.
5.
Berry, D. C. (1987). The problem of implicit knowledge. Expert Systems, 4, 144–151.
6.
Brezovic, C. P., Klein, G., & Thordsen, M. (1987). Decision makingin armored platoon command (Tech. Report MDA903-85-C-0327, U.S. Army Research Institute). Fairborn, OH: Klein Associates, Inc.
7.
Briggs, R. W., & Goldberg, J. H. (1995). Battlefield recognition of armored vehicles. Human Factors, 37, 596–610.
8.
Brown, B. (1989). The taming of an expert: An anecdotal report. SIGART Newsletter, 108, 133–135.
9.
Buckner, V. G. (1997, May). Use of stories in training new employees improves recall. Poster presented at the Annual Convention of the American Psychological Society, Washington, DC.
10.
Calderwood, R., Crandall, B. W., & Klein, G. A. (1987). Expert and novice fireground command decisions (Tech. Report MDA903-85-C-0327, U.S. Army Research Institute). Fairborn, OH: Klein Associates, Inc.
11.
Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Categorization and representations of physics problems by experts and novices. Cognitive Science, 5, 121–152.
12.
Chi, M. T. H., Glaser, R., & Farr, M. J. (Eds.).(1988). The nature of expertise. Mahwah, NJ: Erlbaum.
13.
Chignell, M. H., & Peterson, J. G. (1988). Strategic issues in knowledge engineering. Human Factors, 30, 381–394.
14.
Christensen-Szalanski, J. J. (1993). A comment on applying experimental findings of cognitive biases to naturalistic environments. In G. A. Klein, J. Orasanu, R. Calderwood, & C. E. Zsambok (Eds.), Decision making in action: Models and methods (pp. 252–264). Norwood, NJ: Ablex.
15.
Clancey, W. J. (1993). The epistemology of a rule-based expert system: A framework for explanation. Artificial Intelligence, 20, 215–251.
16.
Cleaves, D. A. (1987). Cognitive biases and corrective techniques: Proposals for improving elicitation procedures for knowledgebased systems. International Journal of Man-Machine Studies, 27, 155–166.
17.
Cook, R. I., Woods, D. D., & McDonald, J. S. (1991). Human performance in anesthesia (Report CSEL91.003). Columbus: Ohio State University, Department of Industrial and Systems Engineering.
18.
Cooke, N. J. (1992). Eliciting semantic relations for empirically derived networks. International Journal of Man-Machine Studies, 37, 721–750.
19.
Cooke, N. J. (1994). Varieties of knowledge elicitation techniques. International Journal of Human-Computer Studies, 41, 801–849.
20.
Cooke, N. J., & Rowe, A. L. (1994). Evaluating mental model elicitation methods. In Proceedings of the Human Factors and Ergonomics Society 38th Annual Meeting (pp. 261–265). Santa Monica, CA: Human Factors and Ergonomics Society.
21.
Crandall, B. (1989). A comparative study of think-aloud and critical decision knowledge elicitation methods. SIGART Newsletter, 108, 144–146.
22.
Crandall, B., & Calderwood, R. (1989). Clinical assessment skills of experienced neonatal intensive care nurses (Report Contract 1-R43-NR01911-01, National Center for Nursing, National Institutes of Health, Bethesda, MD). Fairborn, OH: Klein Associates, Inc.
23.
Crandall, B., & Gamblian, V. (1991). Guide to early sepsis assessment in the NICU. Fairborn, OH: Klein Associates, Inc.
24.
Crandall, B., & Getchell-Reiter, K. (1993). Critical decision method: A technique for eliciting concrete assessment indicators from the “intuition” of NICU nurses. Advances in Nursing Sciences, 16, 42–51.
25.
Crandall, B., & Klein, G. A. (1988a). Critical cues for MI and cardiogenic shock symptoms. Unpublished manuscript.
26.
Crandall, B., & Klein, G. A. (1988b). Key components of MIS performance (Report, Standard Register Company, Inc.). Fairborn, OH: Klein Associates, Inc.
27.
Cullen, J., & Bryman, A. (1988). The knowledge acquisition bottleneck: Time for reassessment?. Expert Systems, 5, 216–225.
28.
Ericsson, K. A., & Simon, H. (1993). Protocol analysis: Verbal reports as data. (2nd ed.). Cambridge: MIT Press.
29.
Evans, J. St. B. T. (1988). The knowledge elicitation problem: A psychological perspective. Behavior and Information Technology, 7, 111–130.
30.
Evans, J. St. B. T. (1989). Bias in human reasoning: Causes and consequences. Mahwah, NJ: Erlbaum.
31.
Fernberger, S. W. (1937). A psychological cycle. American Journal of Psychology, 50, 207–217.
32.
Fischhoff, B. (1989). Eliciting knowledge for analytical representation. IEEE Transactions on Systems, Man and Cybernetics, 19, 448–461.
33.
Fisher, R. P., & Geiselman, R. E. (1992). Memory-enhancing techniques for investigative interviewing: The cognitive interview. Springfield, IL: Charles C Thomas.
34.
Flanagan, J. C. (1954). The critical incident technique. Psychological Bulletin, 51, 327–358.
35.
Fleishman, E. A. (1975). Toward a taxonomy of human performance. American Psychologist, 30, 1127–1149.
36.
Ford, K. M., & Adams-Webber, J. R. (1992). Knowledge acquisition and constructive epistemology. In R. Hoffman (Ed.), The cognition of experts: Psychological research and empirical AI (pp. 121–136). Mahwah, NJ: Erlbaum.
37.
Ford, K. M., & Bradshaw, J. M. (Eds.). (1993). Knowledge acquisition as modeling. New York: Wiley.
38.
Fraser, J. M., Smith, P. J., & Smith, J. W. (1992). A catalog of errors. International Journal of Man-Machine Studies, 37, 265–307.
39.
Gammack, J. G., & Anderson, A. (1990, February). Constructive interaction in knowledge engineering. Expert Systems, 7, 19–26.
40.
Gordon, S. E., & Gill, R. T. (1997). Cognitive task analysis. In C. Zsambok & G. Klein (Eds.), Naturalistic decision making (pp. 131–149). Mahwah, NJ: Erlbaum.
41.
Gordon, S. E., Gill, R. T., & Dingus, T. A. (1987). Designing for the user: The role of human factors in expert system development. AI Applications in Environmental Science, 1, 35–46.
42.
Gordon, S. E., Schmierer, K. A., & Gill, R. T. (1993). Conceptual graph analysis: Knowledge acquisition for instructional system design. Human Factors, 35, 459–481.
43.
Hayes, P. J., Ford, K. M., & Agnew, N. (1994, Fall). On babies and bath water: A cautionary tale. AI Magazine, 15–26.
44.
Hoffman, R. R. (1987, Summer). The problem of extracting the knowledge of experts from the perspective of experimental psychology. AI Magazine, 8, 53–67.
45.
Hoffman, R. R. (1994). Constructivism versus realism or constructivism and realism?. Journal of Experimental and Theoretical Artificial Intelligence, 6, 431–435.
46.
Hoffman, R. R. (1997). A proposed paradigm for the experimental comparison of KA software tools. EVALKA [on-line], Computer Science and Engineering Department, University of New South Wales, Australia. http://www.cse.unsw.edu.au/~timm/pub/eval.
47.
Hoffman, R. R., Feltovich, P. J., & Ford, K. M. (1997). A general framework for conceiving of expertise and expert systems in context. In Feltovich, K. M. Ford, & R. R. Hoffman (Eds.), Expertise in context (pp. 543–580). Cambridge: MIT Press/AAAI Books.
48.
Hoffman, R. R., Shadbolt, N. R., Burton, A. M., & Klein, G. (1995). Eliciting knowledge from experts: A methodological analysis. Organizational Behavior and Human Decision Processes, 62, 129–158.
49.
Holsapple, C. W., & Raj, V. S. (1994). An exploratory study of two KA methods. Expert Systems, 11, 77–87.
50.
Howard, G. S. (1994). Why do people say nasty things about self-reports?. Journal of Organizational Behavior, 15, 399–404.
51.
Hutton, R. J. B., Thordsen, M., & Mogford, R. (1997). Recognition primed decision model in air traffic controller error analysis. In R. S. Jensen & L. A. Rakovan (Eds.), Proceedings of the Ninth International Symposium on Aviation Psychology (pp. 721–726). Columbus: Ohio State University.
52.
Jacob, V. S., Gaultney, L. D., & Salvendy, G. (1986). Strategies and biases in human decision making and their implications for expert systems. Behavior and Information Technology, 5, 119–140.
53.
Jones, S. K., Frisch, D., & Yurak, T. J. (1996). The effect of outcome information on the recall and evaluation of individuals' own decisions. Unpublished manuscript, Ohio State University, Columbus, OH.
54.
Kaempf, G. L., Thordsen, M. L., & Klein, G. (1991). Application of an expertise-centered taxonomy to training decisions (Tech. Report Contract MDA903-91-C-0050, U.S. Army Research Institute). Fairborn, OH: Klein Associates, Inc.
55.
Kaempf, G. L., Wolf, S., Thordsen, M. L., & Klein, G. (1992). Decision making in the AEGIS combat information center (Tech. Report Contract N66001-90-C-6023, Naval Command, Control, and Ocean Surveillance Center). Fairborn, OH: Klein Associates, Inc.
56.
Kidd, A. L. (Ed.). (1987). Knowledge acquisition for expert systems: A practical handbook. New York: Plenum.
57.
Kim, J., & Courtney, J. F. (1988). A survey of knowledge acquisition techniques and their relevance to managerial problem domains. Decision Support Systems, 4, 269–284.
58.
Kinney, W. R., & Uecker, W. C. (1982). Mitigating the consequences of anchoring in auditor judgment. Accounting Review, 57, 55–69.
59.
Klein, G. (1987). Applications of analogical reasoning. Metaphor and Symbolic Activity, 2, 201–218.
60.
Klein, G. (1989). Utility of the critical decision method for eliciting knowledge from expert C debuggers (Report Purchase Order 339404, AT&T Bell Laboratories). Fairborn, OH: Klein Associates, Inc.
61.
Klein, G. (1992). Using knowledge engineering to preserve corporate memory. In R. R. Hoffman (Ed.), The psychology of expertise: Cognitive research and empirical AI (pp. 170–190). Mahwah, NJ: Erlbaum.
62.
Klein, G. (1993a). Naturalistic decision making - Implications for design (Tech. Report Contract SOAR 93-01, Crew System Ergonomics Information Analysis Center). Fairborn, OH: Klein Associates, Inc.
63.
Klein, G. (1993b). The recognition-primed decision model: Looking back, looking forward. In C. Zsambok & G. Klein (Eds.), Naturalistic decision making (pp. 285–292). Mahwah, NJ: Erlbaum.
64.
Klein, G., & Brezovic, C. P. (1986). Design engineers and the design process: Decision strategies and human factors literature. In Proceedings of the Human Factors Society 30th Annual Meeting (pp. 771–775). Santa Monica, CA: Human Factors and Ergonomics Society.
65.
Klein, G. A., Calderwood, R., & Clinton-Cirocco, A. (1986). Rapid decision making on the fire ground. In Proceedings of the Human Factors Society 30th Annual Meeting (pp. 576–580). Santa Monica, CA: Human Factors and Ergonomics Society.
66.
Klein, G., Calderwood, R., & MacGregor, D. (1989). Critical decision method of eliciting knowledge. IEEE Transactions on Systems, Man and Cybernetics, 19, 462–472.
67.
Klein, G., & Hoffman, R. R. (1993). Seeing the invisible: Perceptual-cognitive aspects of expertise. In M. Rabinowitz (Ed.), Cognitive science foundations of instruction (pp. 203–226). Mahwah, NJ: Erlbaum.
68.
Klein, G., Orasanu, J., Calderwood, R., & Zsambok, C. E. (Eds.). (1993). Decision making in action: Models and methods. Norwood, NJ: Ablex.
69.
Klein, G. A., & Thordsen, M. L. (1991). Representing cockpit crew decision making. In R. S. Jensen (Ed.), Proceedings of the Sixth International Symposium on Aviation Psychology (pp. 1026–1031). Columbus: Ohio State University.
70.
Klein, G., & Weitzenfeld, J. (1982). The use of analogues in comparability analysis. Applied Ergonomics, 13, 99–104.
71.
Klinger, D. W., Andriole, S. J., Militello, L. G., Adelman, L., Klein, G., & Gomes, M. E. (1993). Designing for performance: A cognitive systems engineering approach to modifying an AWACS human-computer interface (Tech. Report Contract AL/CF-TR 1993-0093, Air Force Materiel Command). Fairborn, OH: Klein Associates, Inc.
72.
Klinger, D. W., & Gomes, M. G. (1993). A cognitive systems engineering application for interface design. In Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting (pp. 16–20). Santa Monica, CA: Human Factors and Ergonomics Society.
73.
Kuipers, B., & Kassirer, J. P. (1987). Representing expert knowledge. In K. A. Ericsson & J. Smith (Eds.), Toward a general theory of expertise: Prospects and limits (pp. 240–285). Cambridge, England: Cambridge University Press.
74.
Kuipers, B., Moskowitz, A. J., & Kassirer, J. P. (1988). Critical decisions under uncertainty. Cognitive Science, 12, 177–210.
75.
LaFrance, M. (1992). Excavation, capture, collection, and creation: Computer scientists' metaphors for eliciting human expertise. Metaphor and Symbolic Activity, 7, 135–156.
76.
Lesgold, A. M., Rubinson, H., Feltovich, P., Glaser, R., Klopfer, D., & Wang, Y. (1988). Expertise in a complex skill: Diagnosing X-ray pictures. In M. T. H. Chi, R. Glaser, & M. J. Farr (Eds.), The nature of expertise (pp. 311–342). Mahwah, NJ: Erlbaum.
77.
McGraw, K. L., & Seale, M. R. (1988). Knowledge elicitation with multiple experts: Considerations and techniques. Artificial Intelligence Review, 2, 31–44.
78.
Meyer, M. A., & Payton, R. C. (1992). Towards an analysis and classification of approaches to knowledge acquisition from examination of textual metaphor. Knowledge Acquisition, 4, 347–369.
79.
Miller, T. E., Kyne, M., & Lim, L. (1994). Blood banking: The need for training and evaluation (Report Grant 1R43-H149723-01, National Institutes of Health). Fairborn, OH: Klein Associates, Inc.
80.
Miller, T. E., & Lim, L. S. (1993). Using knowledge engineering in the development of an expert system to assist targeteers in assessing battle damage and making weapons decisions for hardened structure targets (Tech. Report Contract DACA39-92-C-0050, U.S. Army Corps of Engineers). Fairborn, OH: Klein Associates, Inc.
81.
Miller, T. E., Wolf, S. P., Thordsen, M. L., & Klein, G. (1992). A decision-centered approach to storyboarding anti-air warfare interfaces (Tech. Report Contract N66001-90-C-6023, Naval Command, Control, and Ocean Surveillance Center). Fairborn, OH: Klein Associates, Inc.
82.
Monk, A. (1985). How and when to collect behavioral data. In A. Monk (Ed.), Fundamentals of human-computer interaction (pp. 69–79). San Diego, CA: Academic.
83.
Mullin, T. M. (1989). Experts' estimation of uncertain quantities and its implications for knowledge acquisition. IEEE Transactions on Systems, Man and Cybernetics, 19, 616–625.
84.
Neale, I. M. (1988). First generation expert systems: A review of knowledge acquisition methodologies. Knowledge Engineering Review, 3, 105–146.
85.
Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231–259.
86.
Noy, N. F., & Hafner, C. D. (1997, Fall). The state of the art in ontology design. AI Magazine, 53–74.
87.
Olson, J. R., & Reuter, H. (1987). Extracting expertise from experts: Methods for knowledge acquisition. Expert Systems, 4, 152–168.
88.
Orr, J. (1985). Social aspects of expertis. (Report). Palo Alto, CA: Xerox PARC.
89.
Patrick, J. (1992). Training: Research and practice. San Diego, CA: Academic.
90.
Pew, R. W., Miller, D. C., & Feehrer, C. E. (1981). Evaluation of proposed control room improvements through analysis of critical operator decisions (Research Report NP-1982). Cambridge, MA: Bolt, Beranek, & Newman, Inc.
91.
Prerau, D. (1989). Developing and managing expert systems: Proven techniques for business and industry. Reading, MA: Addison-Wesley.
92.
Regoczei, S. B., & Hirst, G. (1992). Knowledge and knowledge acquisition in the computational context. In R. R. Hoffman (Ed.), The psychology of expertise: Cognitive research and empirical AI (pp. 12–28). Mahwah, NJ: Erlbaum.
93.
Rolandi, W. G. (1986). Knowledge engineering in practice. AI Expert, 58–62.
94.
Rowe, A. L., & Cooke, N. J. (1995). Measuring mental models: Choosing the right tools for the job. Human Resource Development Quarterly, 6, 243–255.
95.
Rowe, A. L., Cooke, N. J., Hall, E. P., & Halgren, T. L. (1996). Toward an on-line knowledge assessment methodology: Building on the relationship between knowing and doing. Journal of Experimental Psychology: Applied, 2, 31–47.
96.
Sanderson, P. M., Verhapge, A. G., & Fuld, R. B. (1989). Statespace and verbal protocol methods for studying the human operator in process control. Ergonomics, 32, 1343–1372.
97.
Shadbolt, N., & Burton, A. M. (1989). The empirical study of knowledge elicitation techniques. SIGART Newsletter, 108, 21–33.
98.
Shanteau, J. (1984). Some unasked questions about the psychology of expert decision makers. In M. E. El-Hawary (Ed.), Proceedings of the 1984 IEEE Conference on Systems, Man, & Cybernetics (pp. 23–45). New York: Institute of Electrical and Electronics Engineers.
99.
Shanteau, J. (1992). Competence in experts: The role of task characteristics. Organizational Behavior and Human Decision Processes, 53, 252–266.
100.
Shattuck, L., & Woods, D. D. (1994). The critical incident technique 40 years later. In Proceedings of the Human Factors and Ergonomics Society 38th Annual Meeting (pp. 1080–1084). Santa Monica, CA: Human Factors and Ergonomics Society.
101.
Shields, M. D., Solomon, I., & Waller, W. S. (1987). Effects of alternative sample space representations on the accuracy of auditors' uncertainty judgments. Accounting, Organizations, and Society, 12, 375–385.
102.
Sleeman, D., & Shadbolt, N. R. (1996). Report of the ESPRC Workshop on Software Assisted Knowledge Acquisition. Aberdeen, Scotland: University of Aberdeen, Computing Science Department.
103.
Spector, P. E. (1994). Using self-report questionnaires in OB research: A comment on the use of a controversial method. Journal of Organizational Behavior, 15, 385–392.
104.
Taynor, J., Crandall, B., & Wiggins, S. (1987). The reliability of the critical decision method (Tech. Report Contract MDA903-86-C-0170, U.S. Army Research Institute). Fairborn, OH: Klein Associates, Inc.
105.
Thordsen, M. L., Militello, L. G., & Klein, G. A. (1992). Cognitive task analysis of critical team decision making during multiship engagements (Tech. Report Contract F33615-90-C-0005, U.S. Air Force Armstrong Laboratory). Fairborn, OH: Klein Associates, Inc.
106.
Thordsen, M. L., Wolf, S. P., & Crandall, B. (1990). User defined requirements for the JSTARS self defense suite (Report Subcontract A62497L, UNISYS Corporation). Fairborn, OH: Klein Associates, Inc.
107.
Tolcott, M. A., Marvin, F. F., & Lehner, P. E. (1989). Expert decision making in evolving situations. IEEE Transactions on Systems, Man and Cybernetics, 19, 606–615.
108.
Tweney, R. D., Doherty, M. E., & Mynatt, C. R. (Eds.). (1981). On scientific thinking. New York: Columbia University Press.
109.
Vicente, K. J., Christoffersen, K., & Pereklita, A. (1995). Supporting operator problem solving through ecological interface design. IEEE Transactions on Systems, Man and Cybernetics, 25, 529–545.
110.
Waterman, D. A. (1986). A guide to expert systems. Reading, MA: Addison-Wesley.
111.
Weitzenfeld, J. S., & Klein, G. (1979). Analogical reasoning as a discovery logic (Technical Report Contract F49620-79-C-0179, Air Force Office of Scientific Research). Fairborn, OH: Klein Associates, Inc.
112.
Weitzenfeld, J. S., Riedl, T. R., Freeman, J. T., Klein, G., & Musa, J. (1991). Knowledge elicitation for software engineering expertise. In J. E. Tomayko (Ed.), Proceedings of the 1991 Software Engineering Education (SEI) Conference (pp. 283–296). New York: Springer-Verlag.
113.
Wilson, T. D., & Schooler, J. W. (1991). Thinking too much: Introspection can reduce the quality of preferences and decisions. Journal of Personality and Social Psychology, 60, 181–192.
114.
Wood, L. E., Davis, T. C., Clay, S. L., Ford, J. M., & Lammersen, S. (1995). Evaluation of interviewing methods and mediating representations for knowledge acquisition. International Journal of Expert Systems, 8, 1–23.
115.
Wood, L. E., & Ford, K. M. (1993). Structuring interviews with experts during knowledge elicitation. In K. M. Ford & J. M. Bradshaw (Eds.), Knowledge acquisition as modeling (pp. 71–90). New York: Wiley.
116.
Woods, D. D. (1993). Process-tracing methods for the study of cognition outside of the experimental psychology laboratory. In G. A. Klein, J. Orasanu, R. Calderwood, & C. E. Zsambok (Eds.), Decision making in action: Models and methods (pp. 228–251). Norwood, NJ: Ablex.
117.
Zsambok, C. E., Beach, L. R., & Klein, G. (1992). A literature review of analytical and naturalistic decision making. Fairborn, OH: Klein Associates, Inc.
118.
Zsambok, C. E., Kaempf, G. L., Crandall, B., & Kyne, M. (1996). A cognitive model of a prototype training program for OJT providers (Tech. Report Contract MDA903-93-C-0092, U.S. Army Research Institute). Fairborn, OH: Klein Associates, Inc.
119.
Zsambok, C. E., & Klein, G. (Eds.). (1997). Naturalistic decision making. Mahwah, NJ: Erlbaum.
