Cognitive task analysis (CTA) is enjoying growing popularity in both research and practice as a foundational element of instructional design. However, there exists relatively little research exploring its value as a foundation for training through controlled studies. Furthermore, highly individualized approaches to conducting CTA do not permit broadly generalizable conclusions to be drawn from the findings of individual studies. Thus, examining the magnitude of observed effects across studies from various domains and CTA practitioners is essential for assessing replicable effects. This study reports the findings from a meta-analysis that examines the overall effectiveness of CTA across practitioners and settings in relation to other means for identifying and representing instructional content. Overall, the effect of CTA-based instruction is large (Hedges’s g = 0.871). However, effect sizes vary substantially by both CTA method used and training context. Though limited by a relatively small number of studies, the notable effect size indicates that the information elicited through CTA provides a strong basis for highly effective instruction.
AlligerG. M.TannenbaumS. I.BennettW.TraverH.ShotlandA. (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50, 341–358.
2.
*BathalonS.MartinM.DorionD. (2004). Cognitive task analysis, kinesiology and mental imagery: Challenging surgical attrition. Journal of the American College of Surgeons, 199(3), 73.
3.
*BiedermanI.ShiffrarM. M. (1987). Sexing day-old chicks: A case study and expert systems analysis of a difficult perceptual-learning task. Journal of Experimental Psychology: Learning, Memory, and Cognition, 13(4), 640–645.
4.
BlessingS. B.AndersonJ. R. (1996). How people learn to skip steps. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22, 576–598.
5.
*CampbellJ. C. (2010). Employing cognitive task analysis supported instruction to increase medical student and surgical resident performance and self-efficacy (Unpublished doctoral dissertation). University of Southern California, Los Angeles.
6.
CampbellJ. C.TirapelleL.YatesK.ClarkR.InabaK. . . . SullivanM. (2011). The effectiveness of a cognitive task analysis informed curriculum to increase self-efficacy and improve performance for an open cricothyrotomy. Journal of Surgical Education, 68, 403–407.
7.
ChaoC.-J.SalvendyG. (1994). Percentage of procedural knowledge acquired as a function of the number of experts from whom knowledge is acquired for diagnosis, debugging, and interpretation tasks. International Journal of Human-Computer Interaction, 6(3), 221–233.
8.
ClarkR. E. (2009). How much and what type of guidance is optimal for learning from instruction? In TobiasS.DuffyT. M. (Eds.), Constructivist theory applied to instruction: Success or failure? (pp. 158–183). New York, NY: Routledge, Taylor and Francis.
9.
ClarkR. E.EstesF. (1996). Cognitive task analysis. International Journal of Educational Research, 25(2), 403–417.
10.
ClarkR. E.FeldonD.van MerriënboerJ. J. G.YatesK.EarlyS. (2008). Cognitive task analysis. In SpectorJ. M.MerrillM. D.van MerriënboerJ. J. G.DriscollM. P. (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 577–593). New York, NY: Macmillan/Gale.
11.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.) Hillsdale, NJ: Lawrence Erlbaum.
12.
CookeN. J. (1992). Modeling human expertise in expert systems. In HoffmanR. R. (Ed.), The psychology of expertise: Cognitive research and empirical AI (pp. 29–60). Mahwah, NJ: Lawrence Erlbaum.
13.
CookeN. J. (1994). Varieties of knowledge elicitation techniques. International Journal of Human-Computer Studies, 41, 801–849.
14.
CookeN. J. (1999). Knowledge elicitation. In DursoF. T. (Ed.), Handbook of applied cognition (pp. 479–509). New York, NY: John Wiley.
15.
CookeN. J.BreedinS. D. (1994). Constructing naive theories of motion on the-fly. Memory and Cognition, 22, 474–493.
16.
CooperH.HedgesL. V.ValentineJ. C. (Eds.). (2008). The handbook of research synthesis and meta-analysis. New York, NY: Russell Sage.
17.
*CrandallB.CalderwoodR. (1989). Clinical assessment skills of experienced neonatal intensive care nurses (Report Contract 1-R43-NR01911-01). Fairborn, OH: Klein Associates.
18.
CrandallB.Getchell-ReiterK. (1993). Critical decision method: A technique for eliciting concrete assessment indicators from the intuition of NICU nurses. Advances in Nursing Science, 16(1), 42–51.
19.
CrandallB.KleinG.HoffmanR. R. (2006). Working minds: A practitioner’s guide to cognitive task analysis. Cambridge, MA: MIT Press.
20.
*DaRosaD.RogersD. A.WilliamsR. G.HaugeL. S.ShermanH.MurayamaK. . . . DunningtonG. L. (2008). Impact of a structured skills laboratory curriculum on surgery residents’ intraoperative decision-making and technical skills. Academic Medicine, 83(10), S68–S71.
21.
DickW.CareyL.CareyJ. O. (2005). The systematic design of instruction (6th ed.). New York, NY: Pearson.
22.
DunbarK. (2000). How scientists think in the real world: Implications for science education. Journal of Applied Developmental Psychology, 21(1), 49–58.
23.
EricssonK. A.CharnessN. (1994). Expert performance: Its structure and acquisition. American Psychologist, 49(8), 725–747.
24.
FeldonD. F. (2007). The implications of research on expertise for curriculum and pedagogy. Educational Psychology Review, 19(2), 91–110.
25.
FeldonD. F. (2010). Do psychology researchers tell it like it is? A microgenetic analysis of research strategies and self-report accuracy along a continuum of expertise. Instructional Science, 38(4), 395–415.
26.
*FeldonD. F.HurstM. H.ChaoJ.JonesJ. (2009). Evaluation of the CTA-based SNAP training program. Project report generated for the South Carolina Department of Social Services by the Center for Child and Family Studies.
27.
FeldonD. F.StoweK. (2009). A case study of instruction from experts: Why does cognitive task analysis make a difference?Technology, Instruction, Cognition, & Learning, 7, 103–120.
28.
*FeldonD. F.TimmermanB. E.StoweK.ShowmanR. (2010). Translating expertise into effective instruction: The impacts of cognitive task analysis (CTA) on lab report quality and student retention in the biological sciences. Journal of Research on Science Teaching, 47(10), 1165–1185.
29.
GlaserR.ChiM. T. H. (1988). Overview. In ChiM. T. H.GlaserR.FarrM. J. (Eds.), The nature of expertise (pp. xv–xxviii). Mahwah, NJ: Lawrence Erlbaum.
30.
GlaserR.LesgoldA.LajoieS.EastmanR.GreenbergL.LoganD. . . . YengoL. (1985). Cognitive task analysis to enhance technical skills training and assessment (Final Report, Contract No. F41689-8v3-C-0029). Brooks Air Force Base, TX: U.S. Air Force Armstrong Laboratory.
31.
GlassG.PeckhamP.SandersJ. (1972). Consequences of failure to meet assumptions underlying the fixed effects analysis of variance and covariance. Review of Educational Research, 42, 237–288.
32.
*GottS. (1998). Rediscovering learning: Acquiring expertise in real world problem solving tasks (Rep. No. AL/HR-TR-1997-0009). Brooks Air Force Base, TX: U.S. Air Force Armstrong Laboratory.
33.
*GreenR. S. (2008). Cognitive task analyses for life science automation training program design (Unpublished doctoral dissertation). North Carolina State University, Raleigh.
34.
GuimondM. E.SoleM. L.SalasE. (2012). Getting ready for simulation-based training: A checklist for nurse educators. Nursing Education, 32(3), 179–185.
35.
*HallE. P.GottS. P.PokornyR. A. (1995). A procedural guide to cognitive task analysis: The PARI methodology (Rep. No. AL/HR-TR-1995-0108). Brooks Air Force Base, TX: U.S. Air Force Armstrong Laboratory.
36.
HarwellM. R.RubinsteinE. N.HayesW. S.OldsC. C. (1992). Summarizing Monte Carlo results in methodological research: The one- and two-factor fixed effects ANOVA cases. Journal of Educational and Behavioral Statistics, 17(4), 315–339.
37.
HedgesL. V.OlkinI. (1985). Statistical methods for meta-analysis. New York, NY: Academic Press.
38.
JonassenD. H.TessmerM.HannumW. H. (1999). Task analysis methods for instructional design. Mahwah, NJ: Lawrence Erlbaum.
39.
KirschnerP.SwellerJ.ClarkR. E. (2006). Why minimally guided learning does not work: An analysis of the failure of discovery learning, problem-based learning, experiential learning and inquiry-based learning. Educational Psychologist, 41(2), 75–86.
40.
*LajoieS. P. (2003). Individual differences in spatial ability: Developing technologies to increase strategy awareness and skills. Educational Psychologist, 38(2), 115–125.
41.
LeeR. (2005). The impact of cognitive task analysis on performance: A meta-analysis of comparative studies (Unpublished doctoral dissertation). University of Southern California, Los Angeles.
42.
LipseyM. W.WilsonD. B. (2000). Practical meta-analysis. Thousand Oaks, CA: Sage.
43.
LoughnerP.MollerL. (1998). The use of task analysis procedures by instructional designers. Performance Improvement Quarterly, 11(3), 79–101.
44.
*LukerK. R.SullivanM. E.PeyreS. E.ShermanR.GrunwaldT. (2008). The use of a cognitive task analysis–based multimedia program to teach surgical decision making in flexor tendon repair. American Journal of Surgery, 195(1), 11–15.
45.
MalendaM. (2003). In search of the elusive ADDIE model. Performance Improvement, 42(5), 34–36.
46.
*MerrillM. D. (2002). A pebble-in-the-pond model for instructional design. Performance Improvement, 41(7), 39–44.
47.
*ParkC. S.RochlenL. R.YaghmourE.HigginsN.BauchatJ. R.WojciechowskiK. G. . . . McCarthyR. J. (2010). Acquisition of critical intraoperative event management skills in novice anesthesiology residents by using high-fidelity simulation-based training. Anesthesiology, 112(1), 202–211.
48.
ReigeluthC. M. (1983). Current trends in task analysis: The integration of task analysis and instructional design. Journal of Instructional Development, 6(4), 24–30.
49.
*RothE. M.LinL.KerchS.KenneyS. J.SugibayashiN. (2001). Designing a first-of-a-kind group view display for team decision making: A case study. In SalasE.KleinG. (Eds.), Linking expertise and naturalistic decision making (pp. 113–134). Mahwah, NJ: Lawrence Erlbaum.
50.
*SchaafstalA.SchraagenJ. M. (2000). Training of troubleshooting: A structured, task analytical approach. In SchraagenJ.ChipmanS.ShalinV. (Eds.), Cognitive task analysis (pp. 57–70). Mahwah, NJ: Lawrence Erlbaum.
51.
*StaszewskiJ.DavisonA. (2000). Mine detection training based on expert skill. In DubeyA. C.HarveyJ. F.BroachJ. T.DuganR. E. (Eds.), Detection and remediation technologies for mines and mine-like targets V: Proceedings of Society of Photo-Optical Instrumentation Engineers 14th Annual Meeting (Vol. 4038, pp. 90–101). Bellingham, WA: Society of Photo-Optical Instrumentation Engineers.
52.
*SullivanM. E.BrownC. V. R.PeyreS. E.SalimA.MartinM.TowfighS.GrunwaldT. (2007). The use of cognitive task analysis to improve the learning of percutaneous tracheostomy placement. American Journal of Surgery, 193(1), 96–99.
53.
*TirapelleL. A. (2010). The effect of cognitive task analysis based instruction on surgical skills expertise and performance. Los Angeles: University of Southern California.
54.
*van HerzeeleI.AggarwalR.NeequayeS.DarziA.VermassenF.CheshireN. J. (2008). Cognitive training improves clinically relevant outcomes during simulated endovascular procedures. Journal of Vascular Surgery, 48(5), 1223–1230.
55.
van MerriënboerJ. J. G.SwellerJ. (2005). Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review, 17, 147–177.
56.
*VelmahosG. C.ToutouzasK. G.SillinL. F.ChanL.ClarkR. E.TheodorouD.MaupinF. (2004). Cognitive task analysis for teaching technical skills in an inanimate surgical skills laboratory. American Journal of Surgery, 187(1), 114–119.
57.
WeiJ.SalvendyG. (2004). The cognitive task analysis methods for job and task design: Review and reappraisal. Behaviour & Information Technology, 23(4), 273–299.
58.
WoodsD. D.RothE. M. (1988). Cognitive engineering: Human problem solving with tools. Human Factors, 30, 415–430.
59.
YatesK. A.FeldonD. F. (2011). Towards a taxonomy of cognitive task analysis methods for instructional design: Interactions with cognition. Theoretical Issues in Ergonomics Science, 12(6), 472–495.
