The Utility of Event-Based Knowledge Elicitation

Abstract

The purpose of this investigation was to describe and evaluate an event-based knowledge elicitation technique. With this approach experts are provided with deliberate and controlled job situations, allowing investigation of specific task aspects and the comparison of expert responses. For this effort a videotape was developed showing an instructor pilot and student conducting a training mission. Various job situations were depicted in the video to gather information pertinent to understanding team situational awareness. The videotape was shown to 10 instructors and 10 student aviators in the community, and responses to the videotape were collected using a questionnaire at predetermined stop points. Consistent with expectations, the results showed that more experienced respondents (i.e., instructors) identified a richer database of cues and were more likely than students to identify strategies for responding to the situations depicted, providing some empirical evidence for the validity of the event-based technique. This method may serve as a useful knowledge elicitation technique, especially in the later stages of a job analysis when focused information is sought.

Get full access to this article

View all access options for this article.

References

Cannon-Bowers, J. A. , Salas, E. , & Converse, S. A. (1993). Shared mental models in expert team decision making. In N. J. Castellan, Jr. (Ed.), Individual and group decision making: Current issues (pp. 221–246. Mahwah, NJ: Erlbaum.

Converse, S. A., & Kahler, S. E. (1992). Knowledge acquisition and the measurement of shared mental models. Unpublished manuscript, Naval Training Systems Center, Orlando, FL.

Cooke, N. J. (1994). Varieties of knowledge elicitation techniques. International Journal of Human-Computer Studies, 41, 801–849.

Duda, R. O. , & Shortliffe, E. H. (1983). Expert systems research. Science, 220, 261–268.

Endsley, M. R. (1995). Toward a theory of situation awareness in dynamic systems. Human Factors, 37, 32–64.

Ericsson, K. A. , & Lehmann, A. C. (1996). Expert and exceptional performance: Evidence of maximal adaptation to task constraints. Annual Review of Psychology, 47, 273–305.

Fowlkes, J. , Dwyer, D. J. , Oser, R. L. , & Salas, E. (1998). Eventbased approach to training (EBAT). International Journal of Aviation Psychology, 8, 209–221.

Fowlkes, J. E. , Lane, N. E. , Salas, E. , Franz, T. , & Oser, R. (1994). Improving the measurement of team performance: The TARGETs methodology. Military Psychology, 6, 47–61.

Hoffman, R. R. (1987). The problem of extracting the knowledge of experts from the perspective of experimental psychology. AI Magazine, 8, 53–67.

10.

Hoffman, R. R. , Shadbolt, N. R. , Burton, A. M. , & Klein, G. (1995). Eliciting knowledge from experts: A methodological analysis. Organizational Behavior and Human Decision Processes, 62, 129–158.

11.

Johnston, J. H. , Smith-Jentsch, K. A. , & Cannon-Bowers, J. A. (1997). Performance measurement tool for enhancing team decision-making training. In M. T. Brannick , E. Salas , & C. Prince (Eds.), Team performance assessment and measurement: Theory, methods, and applications (pp. 311–327). Mahwah, NJ: Erlbaum.

12.

Rasmussen, J. (1986). Information processing and human machine interaction: An approach to cognitive engineering. New York: Elsevier.

13.

Redding, R. E. , Cannon, J. R. , & Seamster, T. L. (1992). Expertise in air traffic control (ATC): What is it, and how can we train for it? In Proceedings of the Human Factors Society 36th Annual Meeting (pp. 1362–1370). Santa Monica, CA: Human Factors and Ergonomics Society.

14.

Salas, E. , Dickinson, T. L. , Converse, S. , & Tannenbaum, S. I. (1992). Toward an understanding of team performance and training. In R. W. Swezey & E. Salas (Eds.), Teams: Their training and performance (pp. 3–29. Norwood, NJ: Ablex.

15.

Salas, E. , Prince, C. , Baker, D. P. , & Shrestha, L. (1995). Situation awareness in team performance: Implications for measurement and training. Human Factors, 37, 123–136.

16.

Stout, R. J. , Cannon-Bowers, J. A. , & Salas, E. (1994). The role of shared mental models in developing shared situational awareness. In R. D. Gilson , D. J. Garland , & J. M. Koonce (Eds.), Situational awareness in complex systems (pp. 297–304. Daytona Beach, FL: Embry-Riddle Aeronautical University Press.

17.

Zachary, W. W. , Ryder, J. M. , & Hicinbothom, J. H. (in press). Building cognitive task analyses and models of a decision making team in a complex real-time environment. In J. M. Schraagen , S. F. Chipman , & V. Shalin (Eds.), Cognitive task analysis. Mahwah, NJ: Erlbaum.