We propose that considering four categories of task factors can facilitate knowledge elicitation efforts in the analysis of complex cognitive tasks: materials, strategies, knowledge characteristics, and goals. A study was conducted to examine the effects of altering aspects of two of these task categories on problem solving behavior across skill levels: materials and goals. Two versions of an applied engineering problem were presented to expert, intermediate, and novice participants. Participants were to minimize the cost of running a steam generation facility by adjusting steam generation levels and flows. One version was cast in the form of a dynamic, computer-based simulation that provided immediate feedback on flows, costs, and constraint violations, thus incorporating key variable dynamics of the problem context. The other version was cast as a static computer-based model, with no dynamic components, cost feedback, or constraint checking. Experts performed better than the other groups across material conditions, and, when required, the presentation of the goal assisted the experts more than the other groups. The static group generated richer protocols than the dynamic group, but the dynamic group solved the problem in significantly less time. Little effect of feedback was found for intermediates, and none for novices. We conclude that demonstrating differences in performance in this task requires different materials than explicating underlying knowledge that leads to performance. We also conclude that substantial knowledge is required to exploit the information yielded by the dynamic form of the task or the explicit solution goal. This simple model can help to identify the contextual factors that influence elicitation and specification of knowledge, which is essential in the engineering of joint cognitive systems.
Get full access to this article
View all access options for this article.
References
1.
Ahearne, J. (1993). The future of nuclear power. American Scientist, 81, 24–35.
2.
Anderson, J. (1983). The architecture of cognition.Cambridge, MA: Harvard University Press.
3.
Anderson, J. (1985). Cognitive psychology and its implications (2nd ed.). New York: Freeman.
4.
Baddeley, A. (1990). Human memory: Theory and practice.Boston: Allyn & Bacon.
5.
Baron, A., & Galizio, M. (1983). Instructional control of human operant behavior. Psychological Record, 33, 495–520.
6.
Borron, J., Morales, D., & Klahr, P. (1996). Developing and deploying knowledge on a global scale. AI Magazine, 17(4), 65-76.
7.
Brown, J., Collins, A., & Harris, G. (1977). Artificial intelligence and learning strategies (BEN Report 3634, ICAI Report No. 6). Cambridge: Bolt Beranek & Newman, Inc.
8.
Buchanan, B., & Smith, R. (1989). Fundamentals of expert systems. In A. Barr, P. Cohen, & E. Feigenbaum (Eds.), The handbook of artificial intelligence (Vol. 4, pp. 149–192). Reading, MA: Addison-Wesley.
9.
Campbell, C., & Laherrere, J. (1998, March). The end of cheap oil. Scientific American, pp. 78–83.
10.
Card, S. (1996). The human, the computer, the task, and their interaction - Analytic models and use-centered design. In D. Steier & T. Mitchell (Eds.), Mind matters: A tribute to Allen Newell (pp. 259–312. Mahwah, NJ: Erlbaum.
11.
Card, S., Moran, T., & Newell, A. (1983). The psychology of human-computer interaction.Mahwahe, NJ: Erlbaum.
12.
Carroll, J., & Olson, J. (1988). Mental models in human-computer interaction. In M. Helander (Ed.), Handbook of human-computer interaction (pp. 45–65. New York: North-Holland.
13.
Charness, N. (1991). Expertise in chess: The balance between knowledge and search. In K. A. Ericsson & J. Smith (Eds.), Toward a general theory of expertise: Prospects and limits (pp. 39–63. New York: Cambridge University Press.
14.
Clement, J. (1994). Imagistic simulation and physical intuition in expert problem solving. In A. Ram & K. Eiselt (Eds.), Proceedings of the 16th Annual Conference of the Cognitive Science Society (pp. 201–206. Mahwah, NJ: Erlbaum.
15.
deKleer, J. (1977). Multiple representations of knowledge in a mechanics problem solver. In Proceedings of the 5th International Joint Conference on Artificial Intelligence IJCAI-77 (pp. 299–304). San Mateo, CA: Morgan Kaufmann.
16.
Dörner, D., & Schölkopf, J. (1991). Controlling complex systems; or, Expertise as “grandmother's know-how.” In K. A. Ericsson & J. Smith (Eds.), Toward a general theory of expertise: Prospects and limits (pp. 218–239). Cambridge, England: Cambridge University Press.
17.
Ericsson, K. A., & Staszewski, J. (1989). Skilled memory and expertise: Mechanisms of exceptional performance. In D. Klahr & K. Kotovsky (Eds.), Complex information processing: The impact of Herbert A. Simon (pp. 235–267. Mahwah, NJ: Erlbaum.
18.
Eysenck, M., & Keane, M. (1990). Cognitive psychology.Mahwah, NJ: Erlbaum.
19.
Feltovich, P. (1981). Knowledge-based components of expertise in medical diagnosis (LRDC Report PDS-2, September 1981). Pittsburgh: University of Pittsburgh.
20.
Feltovich, P., Ford, K., & Hoffman, R. (Eds.). (1997). Expertise in context.Cambridge, MA: AAAI/MIT Press.
21.
Feltovich, P., Johnson, P., Mohler, J., & Swanson, D. (1984). LCS: The role and development of medical knowledge in diagnostic expertise. In W. Clancey & E. Shortliffe (Eds.), Readings in medical artificial intelligence: The first decade (pp. 275–319. Reading, MA: Addison-Wesley.
22.
Feltovich, P., Spiro, R., & Coulson, R. (1997). Issues of expert flexibility in contexts characterized by complexity and change. In P. Feltovich, K. Ford, & R. Hoffman (Eds.), Expertise in context (pp. 125–146. Cambridge, MA: AAAI/MIT Press.
23.
Gaines, B., & Boose, J. (Eds.). (1988). Knowledge acquisition for knowledge-based systems: Knowledge-based systems, Volume 1.San Diego, CA: Academic.
24.
Gentner, D., & Stevens, A. (Eds.) (1983). Mental models.Mahwah, NJ: Erlbaum.
25.
Hammond, K. (1988). Judgement and decision making in dynamic tasks. Information and Decision Technologies, 14, 3-14.
26.
Hammond, K., Hamm, R., Grassia, J., & Pearson, T. (1987). Direct comparison of the efficacy of intuitive and analytical cognition in expert judgement. IEEE Transactions on Systems, Man and Cybernetics, SMC-17, 753-770.
27.
Hoc, J.-M., Cacciabue, P., & Hollnagel, E. (Eds.). (1995). Expertise and technology: Cognition and human-computer cooperation.Mahwah, NJ: Erlbaum.
28.
Hoffman, R. (1987). The problem of extracting the knowledge of experts. AI Magazine, 8(2), 53-67.
29.
Hoffman, R. (Ed.). (1992). The psychology of expertise.New York: Springer-Verlag.
30.
Hoffman, R., & Deffenbacher, K. (1993). An analysis of the relations between basic and applied psychology. Ecological Psychology, 5(4), 315-352.
31.
Hoffman, R., Feltovich, P., & Ford, K. (1997). A general framework for conceiving of expertise and expert systems in context. In P. Feltovich, K. Ford, & R. Hoffman (Eds.), Expertise in context (pp. 543–580. Cambridge, MA: AAAI/MIT Press.
32.
Hoffrage, U., & Gigerenzer, G. (1996). The impact of information representation on Bayesian reasoning. In G. Cottrell (Ed.), Proceedings of the 18th Annual Conference of the Cognitive Science Society (pp. 126–130. Mahwah, NJ: Erlbaum.
33.
Honeck, R., & Temple, J. (1992). Methaphor, expertise, and a PEST. Metaphor and Symbolic Activity, 7, 237–252.
34.
Huguenard, B., Prietula, M., & Lerch, F. J. (1990, January). Fragility in expertise: a study in reactive scheduling. SIGCHI Bulletin, 21, 36-40.
35.
Jenkins, J. (1979). Four points to remember: A tetrahedral model of memory experiments. In L. Cermak & F. Craik (Eds.), Levels of processing in human memory (pp. 429–446. Mahwah, NJ: Erlbaum.
36.
John, B. (1996). Discussion: Task matters. In D. Steier & T. Mitchell (Eds.), Mind matters: A tribute to Allen Newell (pp. 313–324. Mahwah, NJ: Erlbaum.
37.
Johnson, P., Duran, A., Hassebrock, F., & Moller, J. (1982). Multimethod study of clinical judgement. Organizational Behavior and Human Performance, 30, 201–230.
38.
Johnson, P., Duran, A., Hassebrock, F., Moller, J., Prietula, M., Feltovich, P., & Swanson, D. (1981). Expertise and error in diagnostic reasoning. Cognitive Science, 5, 235–283.
39.
Kempton, W. (1986). Two theories used of heat control. Cognitive Science, 10, 75–91.
40.
Klein, G. (1992). Using knowledge engineering to preserve corporate memory. In R. Hoffman (Ed.), The psychology of expertise (pp. 170–187). New York: Springer-Verlag.
41.
Kuipers, B., & Kassirer, J. (1984). Causal reasoning in medicine: Analysis of a protocol. Cognitive Science, 14, 511–550.
42.
La France, M. (1997). Metaphors for expertise: How knowledge engineers picture human expertise. In P. Feltovich, K. Ford, & R. Hoffman (Eds.), Expertise in context (pp. 163–180. Cambridge, MA: AAAI/MIT Press.
43.
Larkin, J. (1983). The role of problem representation in physics. In D. Gentner & A. Stevens (Eds.), Mental models (pp. 75–98. Mahwah, NJ: Erlbaum.
44.
Larkin, J., & Simon, H. (1987). Why a diagram is (sometimes) worth ten thousand words. Cognitive Science, 11, 65–99.
45.
Lerch, J., Prietula, M., & Kulik, C. (1997). The Turing effect: The nature of trust in expert systems advice. In P. Feltovich, K. Ford, & R. Hoffman (Eds.), Expertise in context (pp. 417–448. Cambridge, MA: AAAI/MIT Press.
46.
Lesgold, A., & Lajoie, S. (1991). Complex problem solving in electronics. In R. Sternberg & P. Frensch (Eds.), Complex problem solving: Principles and mechanisms (pp. 287–316. Mahwah, NJ: Erlbaum.
47.
Lesgold, A., Rubinson, H., Feltovich, P., Glaser, R., Klopfer, D., & Wang, Y. (1988). Expertise in complex skill: Diagnosing X-ray pictures. In M. Chi, R. Glaser, & M. Farr (Eds.), The nature of expertise (pp. 311–342. Mahwah, NJ: Erlbaum.
48.
Lind, M., & Larsen, M. (1995). Planning support and the intentionality of dynamic enviromnents. In J. Hoc, P. Cacciabue, & E. Hollnagel (Eds.), Expertise and technology: Cognition and human-computer interaction (pp. 255–278. Mahwah, NJ: Erlbaum.
49.
Littman, D. (1988). Modelling human expertise in knowledge engineering: Some preliminary observations. In B. Gaines & J. Boose (Eds.), Knowledge acquisition for knowledge-based systems (Vol. 1, pp. 93–104). New York: Academic.
50.
Mayer, R., & Bayman, P. (1981). Psychology of calculator languages: A framework for describing user differences. Communications of the ACM, 24, 511–520.
51.
Newell, A. (1990). Unified theories of cognition.Cambridge, MA: Harvard University Press.
52.
Newell, A., & Simon, H. (1972). Human problem solving.Englewood Cliffs, NJ: Prentice-Hall.
53.
Payne, J., Bettman, J., & Johnson, E. (1988). Adaptive strategy selection in decision making. Journal of Experimental Psychology: Learning, Memory, and Cognition, 14, 534–552.
54.
Pilgrim, C., & Johnston, J. (1988). Laboratory lore and research practices in the experimental analysis of behavior: Issues in instructing subjects. Behavior Analyst, 11, 59–64.
55.
Prerau, D. (1987). Knowledge acquisition in expert systems development. AI Magazine, 8(2), 43-51.
56.
Prietula, M., Feltovich, P., & Marchak, F. (1989). A heuristic framework for assessing factors influencing knowledge acquisition. In R. Blanning & D. King (Eds.), Proceedings of the 22nd Hawaii International Conference on Systems Sciences: Volume 3. Decision support and knowledge-based systems (pp. 419–426). New York: IEEE.
57.
Prietula, M., & March, S. (1991). Form and substance in physical database design: An empirical study. Information Systems Research, 2, 287–314.
58.
Reitman, W. (1965). Cognition and thought.New York: Wiley.
59.
Roy, M., & Lerch, F. J. (1996). Overcoming ineffective mental representations in base-rate problems. Information Systems Research, 7, 233–247.
60.
Simon, H. (1973). The structure of ill-structured problems. Artificial Intelligence, 4, 181–201.
61.
Simon, H. (1996). The sciences of the artificial (3rd ed.). Cambridge, MA: MIT Press.
62.
Vicente, K. (1995). A few implications of an ecological approach to human factors. In J. Flach, P. Hancock, J. Caird, & K. Vicente (Eds.), Global perspectives on the ecology of human-machine systems (pp. 54–67. Mahwah, NJ: Erlbaum.
63.
Vicente, K. (1999). Cognitive work analysis: Toward safe, productive, and healthy computer-based work.Mahwah, NJ: Erlbaum.
64.
Williams, H., Hollan, J., & Stevens, A. (1983). Human reasoning about a simple physical system. In D. Gentner and A. Stevens (Eds.), Mental models (pp. 131–153. Mahwah, NJ: Erlbaum.
65.
Woods, D. (1994). Observations from studying cognitive systems in context. In A. Ram & K. Eiselt (Eds.), Proceedings of the 16th Annual Conference of the Cognitive Science Society (pp. 961–964). Mahwah, NJ: Erlbaum.
66.
Young, R. (1983). Surrogates and mappings: Two kinds of conceptual models for interactive devices. In D. Gentner & A. Stevens (Eds.), Mental models (pp. 35–52. Mahwah, NJ: Erlbaum.
67.
Zhang, J. (1997). The nature of external representations in problem solving. Cognitive Science, 21, 179–217.
68.
Zhang, J., & Norman, D. (1994). Representations in distributed cognitive tasks. Cognitive Science, 18, 87–122
