Objective: The objective of this article is to lay out contributions of human factors to knowledge elicitation (KE) methodology. Background: The background is historical, dating to about 1985, and involves the convergence of expert systems with applied psychology and cognitive psychology. Method: The method is a literature review, focusing on past issues of Human Factors. Results: Human factors researchers have contributed significantly to KE methodology. However, KE methodology “belongs to” a number of communities of practice and has applications that transcend individual disciplines. Conclusion: Knowledge elicitation, thought of as a kind of cognitive task analysis, grows in importance with the increasing use of information technology to form complex sociotechnical work systems and the increasing importance of expertise to knowledge-based organizations. Application: I discuss some open issues for further research and methodological investigation.
Get full access to this article
View all access options for this article.
References
1.
Adelman, L. (1989). Management issues in knowledge elicitation. IEEE Transactions on Systems, Man, and Cybernetics, 19, 483-488.
2.
Allee, V. (1997). The knowledge evolution: Expanding organizational intelligence. Boston: Butterworth-Heinemann .
3.
Bainbridge, L. (1979). Verbal reports as evidence of the process operator's knowledge. International Journal of Man-Machine Studies, 11, 411-436.
4.
Ballas, J.A. (2007). Human centered computing for tactical weather forecasting: An example of the "Moving Target Rule." In R. R. Hoffman (Ed.), Expertise out of context (pp. 316-326). Mahwah, NJ: Erlbaum.
5.
Becerra-Fernandez, I., & Leidner, D. (Eds.). (2008). Knowledge management: An evolutionary view. Upper Saddle River, NJ: Prentice Hall.
6.
Berry, D.C., & Broadbent, D.E. (1984). On the relationship between task performance and associated verbalizable knowledge. Quarterly Journal of Experimental Psychology, 36A, 209-231.
7.
Bransford, J.D., Franks, J.J., Vye, N.J., & Sherwood, R.D. (1989). New approaches to instruction: Because wisdom can't be told. In S. Vosniadou &A. Ortony (Eds.), Similarity and analogical reasoning (pp. 470-497). Cambridge, UK: Cambridge University Press.
8.
Broadbent, D.E., FitzGerald, P., & Broadbent, M.H.P. (1986). Implicit and explicit knowledge in the control of complex systems. British Journal of Psychology , 77, 33-50.
9.
Brooking, A. (1999). Corporate memory: Strategies for knowledge management . London: International Thomson Business Press.
10.
Brown, B. (1989, April). The taming of an expert: An anecdotal report . SIGART Newsletter, pp. 133-135.
11.
Burton, A.M., Shadbolt, N.R., Hedgecock, A.P., & Rugg, G. (1987). A formal evaluation of knowledge elicitation techniques for expert systems: Domain 1. In D. S. Moralee (Ed.), Research and development in expert systems (Vol. 4, pp. 35-46). Cambridge, UK: Cambridge University Press.
12.
Chi, M.T.H., Feltovich, P.J., & Glaser, R. (1981). Categorization and representations of physics problems by experts and novices. Cognitive Science, 5, 121-152.
13.
Chignell, M.H., & Peterson, J.G. (1988). Strategic issues in knowledge engineering. Human Factors, 30, 381-394.
14.
Cooke, N.J. (1985). The elicitation of domain-related ideas: Stage one of the knowledge acquisition process. In C. Ellis (Ed.), Expert knowledge and explanation (pp. 58-75). New York: Wiley.
15.
Cooke, N.J. (1990). Using PATHFINDER as a knowledge elicitation tool: Link interpretation. In R. Schvaneveldt (Ed.), Pathfinder associative networks: Studies in knowledge organization (pp. 227-239). Norwood, NJ: Ablex.
16.
Cooke, N.J. (1992a). Eliciting semantic relations for empirically derived networks. International Journal of Man-Machine Studies , 37, 721-750.
17.
Cooke, N.J. (1992b). Modeling human expertise in expert systems. In R. R. Hoffman (Ed.), The psychology of expertise: Cognitive research and empirical AI (pp. 29-60). New York: Springer Verlag.
18.
Cooke, N.J. (1994). Varieties of knowledge elicitation techniques . International Journal of Human-Computer Studies, 41, 801-849.
19.
Cooke, N.J., & McDonald, J.E. (1986). A formal methodology for acquiring and representing expert knowledge. Proceedings of the IEEE, 74, 1422-1430.
20.
Cooke, N.J., Salas, E., Cannon-Bowers, J.A., & Stout, R.J. (2000). Measuring team knowledge. Human Factors, 42, 151-173.
21.
Crandall, B., & Getchell-Reiter, K. (1993). Critical decision method: A technique for eliciting concrete assessment indicators from the intuition of NICU nurses. Advances in Nursing Science, 16, 42-51.
22.
Crandall, B., Klein, G., & Hoffman R.R. (2006). Working minds: A practitioner's guide to cognitive task analysis. Cambridge, MA: MIT Press.
23.
Cullen, J., & Bryman, A. (1988). The knowledge acquisition bottleneck: Time for a reassessment? Expert Systems, 5, 216-255.
24.
Davenport, T.H., & Prusak, L. (1998). Working knowledge: How organizations manage what they know. Cambridge, MA: Harvard Business School Press.
25.
De Keyser, V. (1997). "Shallow" versus "in-depth" work analysis in human-centered design. In J. Flanagan, T. Huang, P. Jones , & S. Kasif (Eds.), Human-centered systems: Information, interactivity and intelligence (pp. 261-266). Washington, DC: National Science Foundation.
26.
DiBello, L., Missildine, W., & Struttman, M. (in press). Intuitive expertise and empowerment: The long term impact of simulation training on changing accountabilities in a biotech firm. Mind, Culture, and Activity: An International Journal.
27.
Duda, J., Gaschnig, J., & Hart, P. (1979). Model design in the PROSPECTOR consultant system for mineral exploration. In D. Michie (Ed.), Expert systems in the micro-electronic age (pp. 153-167). Edinburgh, UK: Edinburgh University Press.
28.
Ehret, B.D., Gray, W.D., & Kirschenbaum, S.S. (2000). Contending with complexity: Developing and using a scaled world in applied cognitive research. Human Factors, 42, 8-23.
29.
Ericsson, K. A., Charness, N., Feltovich, P. J., & Hoffman, R. R. (Eds.). (2006). Cambridge handbook of expertise and expert performance. New York: Cambridge University Press.
Ericsson, K.A., & Simon, H.A. (1984). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.
32.
Escoto, K.H., Karsh, B.-T., & Beasley, J.W. (2006). Multiple user considerations and their implications in medical error reporting system design. Human Factors, 48, 48-58.
33.
Feigenbaum, E.A., Buchanan, B.G., & Lederberg, J. (1971). On generality and problem solving: A case study using the DENDRAL program. In B. Meltzer & D. Michie (Eds.), Machine intelligence 6 (pp. 165-190). Edinburgh, UK: Edinburgh University Press.
34.
Fisher, A. (2005, March 21). How to battle the coming brain drain . Forbes Magazine, pp. 121-131.
35.
Ford, K. M., & Bradshaw, J. M. (Eds.). (1993). Knowledge acquisition as modeling. New York: Wiley.
36.
Fowlkes, J.E., Salas, E., Baker, D.P., Cannon-Bowers, J.A., & Stout, R.J. (2000). The utility of event-based knowledge elicitation . Human Factors, 42, 24-25.
37.
Gaines, B. (2003). Organizational knowledge acquisition. In C. W. Holsapple (Ed.), Handbook of knowledge management (Vol. 1, pp. 317-347). Berlin: Springer Verlag.
38.
Glaser, R. (1987). Thoughts on expertise. In C. Schooler & W. Schaie (Eds.), Cognitive functioning and social structure over the life course (pp. 81-94). Norwood, NJ: Ablex.
39.
Glaser, R., Lesgold, A., Lajoie, S., Eastman, R., Greenberg, L., Logan, D., et al. (1985). Cognitive task analysis to enhance technical skills training and assessment. Pittsburgh, PA: Learning Research and Development Center, University of Pittsburgh.
40.
Gordon, S.E., Schmierer, K.A., & Gill, R.T. (1993). Conceptual graph analysis: Knowledge acquisition for instructional system design. Human Factors, 35, 459-481.
41.
Hammond, K.R., Hamm, R.M., Grassia, J., & Pearson, T. (1987). Direct comparison of the efficacy of intuitive and analytical cognition in expert judgment. IEEE Transactions on Systems, Man, and Cybernetics, 17, 753-770.
42.
Haughey, D.W. (2000, November). Leveraging Marine Corps experience. Marine Corps Gazette, pp. 45-46.
43.
Hoffman, R.R. (1987, Summer). The problem of extracting the knowledge of experts from the perspective of experimental psychology. AI Magazine, 8, 53-67.
44.
Hoffman, R. R. (Ed.). (1992). The psychology of expertise: Cognitive research and empirical AI. Mahwah, NJ: Erlbaum.
45.
Hoffman, R.R. (1998). How can expertise be defined? Implications of research from cognitive psychology. In R. Williams, W. Faulkner, & J. Fleck (Eds.), Exploring expertise (pp. 81-100). New York: Macmillan.
46.
Hoffman, R.R. (2002). An empirical comparison of methods for eliciting and modeling expert knowledge. In Proceedings of the 46th Meeting of the Human Factors and Ergonomics Society (pp. 482-486). Santa Monica, CA: Human Factors and Ergonomics Society .
47.
Hoffman, R.R., & Coffey, J.W. (2004). Human centered computing: Human factors design principles manifested in a study of weather forecasters. In Proceedings of the 48th Annual Meeting of the Human Factors and Ergonomics Society (pp. 315-319). Santa Monica, CA: Human Factors and Ergonomics Society.
48.
Hoffman, R.R., Coffey, J.W., Ford, K.M., & Carnot, M.J. (2001, October). STORM-LK: A human-centered knowledge model for weather forecasting. In Proceedings of the 45th Annual Meeting of the Human Factors and Ergonomics Society (p. 752). Santa Monica, CA: Human Factors and Ergonomics Society.
49.
Hoffman, R.R., Coffey, J.W., Ford, K.M., & Novak, J.D. (2006). A method for eliciting, preserving, and sharing the knowledge of forecasters. Weather and Forecasting, 21, 416-428.
50.
Hoffman, R.R., Crandall, B., & Shadbolt, N. (1998). Acase study in cognitive task analysis methodology: The critical decision method for the elicitation of expert knowledge. Human Factors, 40, 254-276.
51.
Hoffman, R.R., Feltovich, P.J., & Eccles, D.D. (2007). The cost of knowledge recovery. In Proceedings of the 51st Annual Meeting of the Human Factors and Ergonomics Society (pp. 328-332). Santa Monica, CA: Human Factors and Ergonomics Society.
52.
Hoffman, R.R., & Hanes, L.F. (2003, July-August). The boiled frog problem. IEEE Intelligent Systems, pp. 68-71.
53.
Hoffman, R.R., & Lintern, G. (2006). Eliciting and representing the knowledge of experts . In K. A. Ericsson, N. Charness, P. Feltovich , & R. Hoffman (Eds.), Cambridge handbook on expertise and expert performance (pp. 203-222). New York: Cambridge University Press.
54.
Hoffman, R.R., & Militello, L. (2008). Perspectives on cognitive task analysis. Boca Raton, FL: CRC Press.
55.
Hoffman, R.R., Shadbolt, N., Burton, A.M., & Klein, G.A. (1995). Eliciting knowledge from experts: A methodological analysis. Organizational Behavior and Human Decision Processes , 62, 129-158.
56.
Hoffman, R.R., Trafton, G., & Roebber, P. (2008). Minding the weather: How expert forecasters think . Cambridge, MA: MIT Press.
57.
Hoffman, R. R., & Woods, D. D. (Eds.). (2000). Cognitive task analysis [Special issue]. Human Factors, 42, 1-95.
58.
Josefson, I. (1988). The nurse as engineer: The theory of knowledge in research in the care sector. In B. Goranzon & I. Josefson (Eds.), Knowledge, skill, and artificial intelligence (pp. 19-30). London : Springer Verlag.
59.
Klein, G. (1992). Using knowledge engineering to preserve corporate memory. In R. R. Hoffman (Ed.), The psychology of expertise: Cognitive research and empirical AI (pp. 170-190). Mahwah, NJ: Erlbaum.
60.
Klein, G.A., Calderwood, R., & Clinton-Cirocco, A. (1986). Rapid decision making on the fire ground. In Proceedings of the Human Factors Society 30th Annual Meeting (pp. 576-580). Santa Monica, CA: Human Factors Society.
61.
Klein, G., Calderwood, R., & MacGregor, D. (1989). Critical decision method for eliciting knowledge. IEEE Transactions on Systems, Man, and Cybernetics, 19, 462-472.
LaFrance, M. (1989, April). The quality of expertise: Implications of expert-novice differences for knowledge acquisition. SIGART Newsletter, pp. 6-14.
64.
Lambe, P. (2007). Organizing knowledge: Taxonomies, knowledge and organizational effectiveness . Oxford, England: Chandos Publishing .
65.
Langan-Fox, J., Code, S., & Langfield-Smith, K. (2000). Team mental models: Techniques, methods, and analytic approaches. Human Factors, 42, 242-271.
66.
Militello, L., & Quill, L. (2007). Expert apprentice strategies. In R. R. Hoffman (Ed.), Expertise out of context: Proceedings of the 6th International Conference on Naturalistic Decision Making (pp. 159-176). Boca Raton, FL: Taylor & Francis/ CRC Press.
67.
Mittal, S., & Dym, C.L. (1985, Summer). Knowledge acquisition from multiple experts . The AI Magazine, 6, 32-36.
68.
Mumaw, R.J., Roth, E.M., Vicente, K.J., & Burns, C.M. (2000). There is more to monitoring a nuclear power plant than meets the eye. Human Factors, 42, 36-55.
69.
Nisbett, R.E., & Wilson, T.D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231-259.
70.
Olson, J., & Reuter, H. (1987). Extracting expertise from experts: Methods for knowledge acquisition. Expert Systems, 4, 152-168.
71.
Prietula, M.J., Feltovich, P.J., & Marchak, F. (2000). Factors influencing analysis of complex cognitive tasks: A framework and example from industrial process control. Human Factors, 42, 56-74.
72.
Regoczei, S.B., & Hirst, G. (1992). Knowledge and knowledge acquisition in the computational context. In R. R. Hoffman (Ed.), The psychology of expertise: Cognitive research and empirical AI (pp. 12-28). Mahwah, NJ: Erlbaum.
73.
Rodi, L.L., Pierce, J.A., &Dalton, R.E. (1989, April). Putting the expert in charge: Graphical knowledge acquisition for fault diagnosis and repair. SIGART Newsletter, pp. 56-62.
74.
Salas, E., Cooke, N.J., & Rosen, M.A. (2008, this issue). On teams, teamwork, and team performance: Discoveries and developments. Human Factors, 50, 540-547.
75.
Salter, W.J. (1988). Human factors in knowledge acquisition. In M. Helander (Ed.), Handbook of human-computer interaction (pp. 957-968). Amsterdam: North-Holland.
76.
Schaafstal, A., Schraagen, J.M., & van Berlo, M. (2000). Cognitive task analysis and innovation in training: The case of structured troubleshooting. Human Factors, 42, 75-86.
77.
Schweickert, R., Burton, A.M., Taylor, N.K., Corlett, E.N., Shadbolt, N.R., &Hedgecock,A.P. (1987). Comparing knowledge elicitation techniques: A case study. Artificial Intelligence Review, 1, 245-253.
78.
Shadbolt, N.R., & Burton, A.M. (1990). Knowledge elicitation techniques: Some experimental results. In K. L. McGraw & C. R. Westphal (Eds.), Readings in knowledge acquisition (pp. 21-33). New York: Ellis Horwood.
