Complex domains require cognitive work for which current approaches to training may be ill-suited. To improve training for cognitive work, Klein and Baxter have proposed Cognitive Transformation Theory (CTT), a learning theory that characterizes sensemaking processes as essential to the development of expertise. The objectives of this research were to compare CTT with the instructional strategies of two expert air traffic control instructors to evaluate the relevance of CTT’s four teaching practices, propose refinements to CTT, and identify potential instructional strategies to serve as guidance for the application of CTT. Data were collected using cognitive task analysis methods, including course observation, artifact examination, and knowledge elicitation with two instructors and seven of their students. Data were coded using categories derived from theory and patterns emergent within the data. Results suggest that many of the instructional strategies used were consistent with the teaching practices of CTT and that learning was aligned with the active sensemaking claims of CTT. An integrated set of instructional strategies and a few refinements to CTT are advanced to further its application to training in complex domains. Although this set of strategies may benefit current training practices, further research is needed to evaluate their effectiveness.
AndersonJ. R. (1983). The architecture of cognition. Cambridge, MA: Harvard University Press.
2.
ArthurW.Jr.BennettW.Jr.EdensP. S.BellS. T. (2003). Effectiveness of training in organizations: A meta-analysis of design and evaluation features. Journal of Applied psychology, 88(2), 234-245.
3.
Auer-SrnkaK.KoeszegiS. (2007). From words to numbers: How to transform qualitative data into meaningful quantitative results. Schmalenbach Business Review, 59, 29-57.
4.
BanerjeeM.CapozzoliM.McSweeneyL.StinhaD. (1999). Beyond kappa: A review of interrater agreement measures. Canadian Journal of Statistics, 27, 3-23.
5.
BergerA.KofmanO.LivnehU.HenikA. (2007). Multidisciplinary perspectives on attention and the development of self-regulation. Progress in Neurobiology, 82, 256-287.
6.
BransfordJ. D.BrownA. L.CockingR. R. (2000). How people learn: Brain, mind, experience, and school: Expanded edition. Washington, DC: National Academy Press.
7.
BransfordJ. D.FranksJ. J.MorrisC. D.SteinB. S. (1979). Some general constraints on learning and memory research. In CermackL. S.CraikF. I. M. (Eds.), Levels of processing in human memory (pp. 331-354). Hillsdale, NJ: Erlbaum.
8.
BransfordJ. D.FranksJ. J.VyeN. J.SherwoodR. D. (1989). New approaches to instruction: Because wisdom can’t be told. In VosniadouS.OrtonyA. (Eds.), Similarity and analogical reasoning (pp. 470-497). Cambridge, UK: Cambridge University Press.
9.
BransfordJ. D.SchwartzD. L. (2009). It takes expertise to make expertise: Some thoughts about why and how and reflections on the themes in Chapters 15-18. In Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments (pp. 432-438). Cambridge, UK: Cambridge University Press.
10.
BraunV.ClarkeV. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101.
11.
ButcherK. R.SumnerT. (2011). Self-directed learning and the sensemaking paradox. Human-Computer Interaction, 26(1-2), 123-159.
12.
ChaseW. G.SimonH. A. (1973). Perception in chess. Cognitive Psychology, 4, 55-81.
13.
ChiM. T. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 6(3), 271-315.
14.
ChiM. T. (2006). Two approaches to the study of experts’ characteristics. In The Cambridge handbook of expertise and expert performance (pp. 21-30). Cambridge, UK: Cambridge University Press.
15.
ChiM. T. H. (2008). Three types of conceptual change: Belief revision, mental model transformation, and categorical shift. In VosniadouS. (Ed.), Handbook of research on conceptual change (pp. 61-82). Hillsdale, NJ: Erlbaum.
16.
CraikK. (1943). The nature of explanation. Cambridge, UK: Cambridge University Press.
17.
CrandallB.KleinG.HoffmanR. (2006). Working minds: A practitioner’s guide to cognitive task analysis. Cambridge, MA: The MIT Press.
18.
DeweyJ. (1938). Experience and education. New York: MacMillan.
19.
DignathC.BüttnerG. (2008). Components of fostering self-regulated learning among students. A meta-analysis on intervention studies at primary and secondary school level. Metacognition and Learning, 3(3), 231-264.
20.
DonovanJ. J.RadosevichD. J. (1999). A meta-analytic review of the distribution of practice effect: Now you see it, now you don’t. Journal of Applied Psychology, 84(5), 795-805.
21.
DursoF. T.ManningC. A. (2008). Air traffic control. Review of Human Factors and Ergonomics, 4(1), 195-244. doi: 10.1518/155723408X342853
22.
EricssonK. A. (2006). The influence of experience and deliberate practice on the development of superior expert performance. In EricssonK. A.CharnessN.FeltovichP. J.HoffmanR. R. (Eds.), Expertise and expert performance (pp. 683-703). New York: Cambridge University Press.
23.
EricssonK. A. (Ed.). (2009). Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments. Cambridge, UK: Cambridge University Press.
24.
EricssonK.CharnessN. E.FeltovichP. J.HoffmanR. R. (2006). The Cambridge handbook of expertise and expert performance. Cambridge, UK: Cambridge University Press.
25.
EricssonK. A.KintschW. (1995). Long-term working memory. Psychological Review, 102(2), 211-245.
26.
EricssonK. A.KrampeR. T.Tesch-RomerC. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100(3), 363-406.
27.
EricssonK. A.WardP. (2007). Capturing the naturally occurring superior performance of experts in the laboratory: Toward a science of expert and exceptional performance. Current Directions in Psychological Science, 16(6), 346-350.
28.
FeltovichP. J.SpiroR. J.CoulsonR. L. (1993). Learning, teaching, and testing for complex conceptual understanding. In FrederiksenN.MislevyR. J.BejarI. I. (Eds.), Test theory for a new generation of tests (pp. 181-217). Mahwah, NJ: Erlbaum.
29.
FeltovichP. J.SpiroR. J.CoulsonR. L. (2001). Learners’ (mis)understanding of important and difficult concepts: A challenge to smart machines. In ForbusK.FeltovichP. J. (Eds.), Smart machines in education (pp. 349-376). Cambridge, MA: MIT Press.
30.
FioreS. M.CuevasH. M.OserR. L. (2003). A picture is worth a thousand connections: The facilitative effects of diagrams on mental model development and task performance. Computers in Human Behavior, 19, 185-199.
31.
FioreS.Vogel-WalcuttJ. J. (2010). Making metacognition explicit: Developing a theoretical foundation for metacognitive prompting during scenario-based training. Proceeding of the Human Factors and Ergonomics Society 54th Annual Meeting, 54(27), 2233-2237.
32.
FlavellJ. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American Psychologist, 34(10), 906-911.
33.
FowlkesJ.NormanJ. W.SchatzS.StaglK. C. (2009). Contrasting cases: A strategy for advanced learning using simulation-based training. Proceedings of the Human Factors and Ergonomics Society 53rd Annual Meeting, 53(26), 1935-1938.
34.
GentnerD.StevensA. L (1983). Mental models. Hillsdale, NJ: Erlbaum.
35.
GobetF.SimonH. A. (1996). Templates in chess memory: A mechanism for recalling several boards. Cognitive Psychology, 31, 1-40.
36.
GobetF.SimonH. A. (1998). Expert chess memory: Revisiting the chunking hypothesis. Memory, 6, 225-255.
37.
GoodmanJ. S.WoodR. E.HendrickxM. (2004). Feedback specificity, exploration, and learning. Journal of Applied Psychology, 89(2), 248.
38.
HartmanH.SternbergR. J. (1993). A broad BACEIS for improving thinking. Instructional Science, 21(5), 400-425.
39.
HatanoG.InagakiK. (1986). Two courses of expertise. In StevensonH. A. H.HakutaK. (Eds.), Child development and education in Japan (pp. 262-272). New York: Freeman.
HoffmanR. R.WardP.FeltovichP. J.DiBelloL.FioreS. M.AndrewsD. H. (2014). Accelerated expertise: Training for high proficiency in a complex world. New York: Psychology Press.
42.
HutchinsS. G.PirolliP. L.CardS. K. (2007). What makes intelligence analysis difficult? A cognitive task analysis. In HoffmanR. R. (Ed.), Expertise out of context: Proceedings of the Sixth International Conference on Naturalistic Decision Making (pp. 281-316). New York: Erlbaum.
43.
JohnsonR. B. (1997). Examining the validity structure of qualitative research. Education, 118(2), 282-292.
44.
Johnson-LairdP. N. (1983). Mental models: Towards a cognitive science of language, inference, and consciousness. Cambridge, MA: Harvard University Press.
45.
Johnson-LairdP. N. (2004). The history of mental models. In ManketowK.ChungM. C. (Eds.), Psychology of reasoning: Theoretical and historical perspectives (pp. 179-212). New York: Psychology Press.
46.
KintschW. (1988). The role of knowledge in discourse comprehension: A construction-integration model. Psychological Review, 95(2), 163-182.
47.
KintschW. (1998). Comprehension: A paradigm for cognition. Cambridge, UK: Cambridge University Press.
48.
KintschW. (2005). An overview of top-down and bottom-up effects in comprehension: The CI perspective. Discourse Processes, 39(2-3), 125-128.
49.
KirschenbaumS. S.TraftonJ. G.PrattE. (2007). Comparative cognitive task analysis. In HoffmanR. R. (Ed.), Expertise out of context: Proceedings of the Sixth International Conference on Naturalistic Decision Making (pp. 327-336). New York: Erlbaum.
50.
KleinG. A. (1989). Recognition-primed decisions. In RouseW. B. (Ed.), Advances in man- machine systems research (Vol. 5, pp. 47-92). Greenwich, CT: JAI Press, Inc.
51.
KleinG. A. (1993). A recognition primed (RPD) model of rapid decision making. In KleinG. A.OrasnuJ.CalderwoodR.ZsambokC. E. (Eds.), Decision making in context (pp. 138-147). Norwood, NJ: Ablex.
52.
KleinG. (1998). Sources of power. Cambridge, MA: MIT Press.
53.
KleinG.BaxterH. C. (2006). Cognitive transformation theory: Contrasting cognitive and behavioral learning. In Proceedings of the Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) (pp. 1-9). Arlington, VA: NTSA.
54.
KleinG.BaxterH. C. (2009). Cognitive transformation theory: Contrasting cognitive and behavioral learning. In CohnJ. V.SchmorrowD.NicholsonD. (Eds.), The PSI handbook of virtual environments for training and education: Developments for the military and beyond. Vol 1. Learning, requirements, and metrics (pp. 50-64). Westport, CT: Praeger Security International.
55.
KleinG. A.CalderwoodR.Clinton-CiroccoA. (1986). Rapid decision making on the fireground. Proceedings of the Human Factors and Ergonomics Society 30th Annual Meeting, 1, 576-580.
56.
KleinG.MoonB.HoffmanR. R. (2006a). Making sense of sensemaking 1: Alternative perspectives. IEEE Intelligent Systems, 21(4), 70-73.
57.
KleinG.MoonB.HoffmanR. R. (2006b). Making sense of sensemaking 2: A macrocognitive model. IEEE Intelligent Systems, 21(5), 87-92.
58.
KleinG.PhillipsJ. K.RallE.PelusoD. A. (2007). A data/frame theory of sensemaking. In HoffmanR. R. (Ed.), Expertise out of context: Proceedings of the 6th International Conference on Naturalistic Decision Making (pp. 113-155). Mahwah, NJ: Lawrence Erlbaum.
59.
KleinG.PliskeR. M.CrandallB.WoodsD. (2005). Problem detection. Cognition, Technology, and Work, 7, 14-28.
KleinG.WigginsS.DominguezC. O. (2010). Team sensemaking. Theoretical Issues in Ergonomics Science, 11(4), 304-320.
62.
KleinG.WolfS. (1998). The role of leverage points in option generation. IEEE Transactions on Systems, Man and Cybernetics: Applications and Reviews, 28(1), 157-160.
63.
KolbD. A. (1984). Experiential learning: Experience as the source of learning and development. Englewood Cliffs, NJ: Prentice Hall.
64.
KozlowskiS. W.IlgenD. R. (2006). Enhancing the effectiveness of work groups and teams. Psychological Science in the Public Interest, 7(3), 77-124.
65.
KraigerK.FordJ. K.SalasE. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78(2), 311-328.
66.
LairdJ. E.NewellA.RosenbloomP. S. (1987). Soar: An architecture for general intelligence. Artificial Intelligence, 33(1), 1-64.
67.
LewandowskyS.DunnJ. C.KirsnerK.RandallM. (1997). Expertise in the management of bush fires: Training and decision support. Australian Psychologist, 32(3), 171-177.
68.
MarshallS. P. (1995). Schemas in problem solving. New York: Cambridge University Press.
69.
MathanS. A.KoedingerK. R. (2005). Fostering the intelligent novice: Learning from errors with metacognitive tutoring. Educational Psychologist, 40(4), 257-265.
70.
MayerR. E. (2005). Cognitive theory of multimedia learning. In The Cambridge Handbook of Multimedia Learning (pp. 31-48). New York: Cambridge University Press.
MilitelloL. G.HuttonR. J. (1998). Applied Cognitive Task Analysis (ACTA): A practitioner’s toolkit for understanding cognitive task demands. Ergonomics, 41(11), 1618-1641.
73.
O’BrienE. J.AlbrechtJ. E. (1992). Comprehension strategies in the development of a mental model. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18(4), 777.
ReddingR. E. (1989). Trainers teaching thinking skills: Applications of recent research in metacognition to training. Proceedings of the Human Factors Society 33rd Annual Meeting, 33(19), 1353-1357.
76.
SalasE.Cannon-BowersJ. A. (2001). The science of training: A decade of progress. Annual Review of Psychology, 52(1), 471-499.
77.
SchwartzD. L.BransfordJ. D. (1998). A time for telling. Cognition and Instruction, 16, 475-522.
78.
SchwartzD. L.BransfordJ. D.SearsD. (2005). Efficiency and innovation in transfer. In MestreJ. (Ed.), Transfer of learning from a modern multidisciplinary perspective (pp. 1-51). Greenwich, CT: Information Age Publishing.
79.
SeamsterT. L.ReddingR. E.CannonJ. R.RyderJ. M.PurcellJ. A. (1993). Cognitive task analysis of expertise in air traffic control. International Journal of Aviation Psychology, 3(4), 257-283.
80.
ShiffrinR. M.SchneiderW. (1977). Controlled and automatic human information processing: II. Perceptual learning, automatic attending and a general theory. Psychological Review, 84(2), 127-190.
81.
ShuteV. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153-189.
82.
SieckW. R.KleinG.PelusoD. A.SmithJ. L.Harris-ThompsonD. (2007). Focus—A model of sensemaking (Technical Report No. 1200). Retrieved from www.au.af.mil/au/awc/awcgate/army/tr1200.pdf
83.
SimonH. A.ChaseW. G. (1973). Skill in chess. American Scientist, 61, 394-403.
84.
SitzmannT.ElyK. (2011). A meta-analysis of self-regulated learning in work-related training and educational attainment: What we know and where we need to go. Psychological Bulletin, 137(3), 421-442.
85.
Smith-JentschK. A.ZeisigR. L.ActonB.McPhersonJ. A. (1998). Team dimensional training: A strategy for guided team self-correction. In Cannon-BowersJ. A.SalasE. E. (Eds.), Making decisions under stress: Implications for individual and team training (pp. 271-297). Washington, DC: American Psychological Association.
86.
SottilareR.GoldbergB. (2012). Designing adaptive computer-based tutoring systems to accelerate learning and facilitate retention. Cognitive Technology, 17(1), 19-33.
87.
SternbergR. J. (1998). Metacognition, abilities, and developing expertise: What makes an expert student?Instructional Science, 26(1-2), 127-140.
88.
Tofel-GrehlC.FeldonD. F. (2013). Cognitive task analysis-based training: A meta-analysis of studies. Journal of Cognitive Engineering and Decision Making, 7(3), 294-304.
89.
Van MerriënboerJ. J. G.JelsmaO.PaasF. G. W. C. (1992). Training for reflective expertise: A four component instructional design model for complex cognitive skills. Educational Technology Research and Development, 49(2), 23-43.
90.
WiltshireT. J. (2012). Exploration of sensemaking in the education of novices to the complex cognitive work domain of air traffic control. Unpublished master’s thesis, Embry-Riddle Aeronautical University, Daytona Beach, FL.
91.
WulfG.McConnelN.GartnerM.ScharwzA. (2002). Enhancing the learning of sport skills through external focus feedback. Journal of Motor Behavior, 34(2), 171-182.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
