Objective: This review provides the state of crew resource management (CRM) training evaluations since the E. Salas, C. S. Burke, C. A. Bowers, and K. A. Wilson (2001) review and extends it to areas beyond aviation cockpits. Some critical evaluation needs in CRM training are also covered. Background: Because of the purported success of CRM training in aviation, other high-consequence domains have begun to implement CRM training for their workforces. However, the true impact of CRM training in aviation and these other domains has yet to be determined. Method: Using D. L. Kirkpatrick's (1976) framework for evaluating training (i.e., reactions, learning, behavior, and organizational impact), we reviewed 28 published accounts of CRM training to determine its effectiveness within aviation, medicine, offshore oil production and maintenance, shipping/maritime, and nuclear power domains. Results: Findings indicate that CRM training generally produced positive reactions from trainees; however, the impact of training on learning and behavioral changes suggest mixed results across and within domains. Furthermore, and as was found by Salas, Burke, et al. in 2001, we cannot ascertain whether CRM has had an impact on the organization's bottom line (i.e., safety). Conclusion: Based on the results, there are several critical needs that the CRM training community must address before CRM training can have the desired impact on safety: a mandate, access to data, and resources. Application: As CRM training expands to organizations beyond aviation, it is critical that its impact be understood such that it can be improved and achieve the intended results.
Get full access to this article
View all access options for this article.
References
1.
Alliger, G. M., Tannenbaum, S. I., Bennett, W., Jr., & Traver, H. (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50, 341–358.
2.
Besco, R. O. (1995). The potential contributions and scientific responsibilities of aviation psychologists. In N. Johnston, R. Fuller, & N. McDonald (Eds.), Aviation psychology: Training and selection: Proceedings of the 21st Conference of the European Association for Aviation Psychology (Vol. 2, pp. 141–148). Aldershot, UK: Avebury Aviation.
3.
Besco, R. O. (1997). The need for operational validation of human relations-centered CRM training assumptions. In R. S. Jensen & L. A. Rakovan (Eds.), Proceedings of the 9th International Symposium on Aviation Psychology (pp. 536–540). Columbus: Ohio State University.
4.
Besco, R. O. (1998). Crew resource management training: What to teach and how to teach it! Retrieved September 21, 2000, from http://www.ppi.aero/commentary/crmcritique.htm
5.
Bowers, C. A., & Jentsch, F. (2001). Use of commercial, off-the-shelf, simulations for team research. In E. Salas (Ed.), Advances in human performance (Vol. 1, pp. 293–317). Amsterdam: Elsevier Science.
6.
Brun, W., Eid, J., Jihnsen, B. H., Ekornas, B., Laberg, J. C., & Kob-beltvedt, T. (2000). Shared mental models and task performance: Studying the effects of a crew and bridge resource management training program (Project Report: 1 2001). Bergen, Norway: Militaer Psykologi og Ledelse.
7.
Davis, D. D., Bryant, J., Liu, Y., Tedrow, L., & Say, R. (2003). National culture, team behavior and error management in U.S. and Chinese simulated aircrews. In R. S. Jensen (Ed.), Proceedings of the 12th International Symposium on Aviation Psychology (pp. 273–277). Columbus: Ohio State University.
8.
Edkins, G. D. (2002). A review of the benefits of aviation human factors training. Human Factors and Aerospace Safety, 2, 201–216.
9.
Ellis, C. & Hughes, G. (1999). Use of human patient simulation to teach emergency medicine trainees advanced airway skills. Journal of Accident Emergency Medicine, 16, 395–399.
10.
Fisher, J., Phillips, E., & Mather, J. (2000). Does crew resource management training work?Air Medical Journal, 19, 137–139.
11.
Flin, R., Martin, L., Goeters, K., Hormann, H., Amalberti, R., Valot, C., et al. (2003). Development of the NOTECHS (non-technical skills) system for assessing pilot's CRM skills. Human Factors and Aerospace Safety, 3, 95–117.
12.
Fonne, V. M., & Fredriksen, O. K. (1995). Resource management and crew training for HSV-navigators. In R. S. Jensen & L. A. Rakovan (Eds.), Proceedings of the 8th International Symposium on Aviation Psychology (pp. 585–590). Columbus: Ohio State University.
13.
Gaba, D. M., Howard, S. K., Fish, K. J., Smith, B. E., & Sowb, Y. A. (2001). Simulation-based training in anesthesia crisis resource management (ACRM): A decade of experience. Simulation and Gaming, 32, 175–193.
14.
Gaba, D. M., Howard, S. K., Flanagan, B., Smith, B. E., Fish, K. J., & Botney, R. (1998). Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology, 89, 8–18.
15.
Goeters, K. M. (2002). Evaluation of the effects of CRM training by the assessment of non-technical skills under LOFT. Human Factors and Aerospace Safety, 2, 71–86.
16.
Grubb, G., Crossland, N., & Katz, L. (2002). Evaluating and delivering the U.S. Army aircrew coordination training enhancement (ACTE) program. In Proceedings of the Interservice/Industry Training, Simulation and Education Conference (pp. 1143–1149). Arlington, VA: National Training Systems Association.
17.
Grubb, G., & Morey, J. C. (2003). Enhancement of the U.S. Army aircrew coordination training (ACT) program. In R. S. Jensen (Ed.), Proceedings of the 12th International Symposium on Aviation Psychology (pp. 446–452). Columbus: Ohio State University.
18.
Halamek, L. P., Kaegi, D. M., Sowb, Y. A., Smith, B. C., Smith, B. E., & Howard, S. K. (2000). Time for a new paradigm in pediatric medical education: Teaching neonatal resuscitation in a simulated delivery room environment. Pediatrics, 106, Article e45. Retrieved November 19, 2003, from http://pediatrics.aappublications.org/cgi/reprint/106/4/e45
19.
Helmreich, R. L. (1984). Cockpit management attitudes. Human Factors, 26, 583–589.
20.
Helmreich, R. L., Merritt, A. C., & Wilhelm, J. A. (1999). The evolution of crew resource management training in commercial aviation. International Journal of Aviation Psychology, 9, 19–32.
21.
Holzman, R. S., Cooper, J. B., Gaba, D. M., Philip, J. H., Small, S. D., & Feinstein, D. (1995). Anesthesia crisis resource management: Real-life simulation training in operating room crises. Journal of Clinical Anesthesia, 7, 675–687.
22.
Howard, S. K., Gaba, D. M., Fish, K. J., Yang, G., & Sarnquist, F. H. (1992). Anesthesia crisis resource-management training: Teaching anesthesiologists to handle critical incidents. Aviation, Space, and Environmental Medicine, 63, 763–770.
23.
iGardi, T., Christensen, U. C., Jacobsen, J., Jensen, P. F., & Ording, H. (2001). How do anaesthesiologists treat malignant hyperthermia in a full-scale anaesthesia simulator?Acta Anaesthesiologica Scandinavica, 45, 1032–1035.
24.
Jacobsen, J., Lindekaer, A. L., Ostergaard, H. T., Nielsen, K., Ostergaard, D., Laub, M., et al. (2001). Management of anaphylactic shock evaluated using a full-scale anaesthesia simulator. Acta Anaesthesiologica Scandinavica, 45, 315–319.
25.
Karp, M. R., Condit, D., & Nullmeyer, R. T. (1999). Cockpit/crew resource management for single-seat fighter pilots. In Proceedings of the Interservice/Industry Training, Simulation, and Education Conference [CD-ROM]. Arlington, VA: National Training Systems Association.
26.
Karp, M. R., Condit, D., & Nullmeyer, R. T. (2000). Survey of cockpit/crew resource management for F-16 pilots (Rep. No. AFRL-HE-AZ-TR-1999–0253). Mesa, AZ: Air Force Research Laboratory.
27.
Katz, L. (2003). Army CRM training: Demonstration of a prototype computer-based program. In Proceedings of the 12th International Symposium on Aviation Psychology (pp. 648–650). Columbus: Ohio State University.
28.
Kirkpatrick, D. L. (1976). Evaluation of training. In R. L. Craig (Ed.), Training and development handbook: A guide to human resources development (pp. 18.1–18.27). New York: McGraw-Hill.
29.
Komich, J. (1997). CRM training: Which crossroads to take now? In R. S. Jensen & L. A. Rakovan (Eds.), Proceedings of the 9th International Symposium on Aviation Psychology (pp. 541–546). Columbus: Ohio State University.
30.
Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, 311–328.
31.
Kurrek, M. M. & Fish, K. J. (1996). Anaesthesia crisis resource management training: An intimidating concept, a rewarding experience. Canadian Journal of Anaesthesia, 43, 430–434.
32.
Lainos, J. S., & Nikolaidis, E. D. (2003). Presuppositions for the effective introduction of changes to the aviation safety culture, through CRM training program: The case study of the Olympic Airways. In R. S. Jensen (Ed.), Proceedings of the 12th International Symposium on Aviation Psychology (pp. 694–699). Columbus: Ohio State University.
33.
Morey, J. C., Simon, R., Jay, G. D., Wears, R. L., Salisbury, M., Dukes, K. A., et al. (2002). Error reduction and performance improvement in the emergency department through formal teamwork training: Evaluation results of the MedTeams project. Health Services Research, 37, 1553–1581.
34.
Nullmeyer, R., Spiker, V. A., Wilson, D., & Deen, G. (2003). Key crew resource management behaviors underlying C-130 aircrew performance. In Proceedings of the Interservice/Industry Training, Simulation, and Education Conference [CD-ROM]. Arlington, VA: National Training Systems Association.
35.
O'Connor, P., & Flin, R. (2003). Crew resource management training for offshore oil production teams. Safety Science, 41, 591–609.
36.
O'Connor, P., Flin, R., Fletcher, G., & Hemsley, P. (2002). Methods used to evaluate the effectiveness of flightcrew CRM training in the UK aviation industry. Human Factors and Aerospace Safety, 2, 235–255.
37.
O'Connor, P., Flin, R., Fletcher, G., & Hemsley, P. (2003). Methods used to evaluate the effectiveness of flightcrew CRM training in the UK aviation industry (Civil Aviation Authority Paper 2002/05). Retrieved October 19, 2003, from http://www.caa.co.uk/docs/33/CAPAP2002_05.PDF
38.
O'Donnell, J., Fletcher, J., Dixon, B., & Palmer, L. (1998). Planning and implementing an anesthesia crisis resource management course for student nurse anesthetists. CRNA: The Clinical Forum for Nurse Anesthetists, 9, 50–58.
39.
Orasanu, J., Murray, L., Rodvold, M. A., & Tyzzer, L. K. (1999). Has CRM succeeded too well? Assertiveness on the flight deck. In R. S. Jensen (Ed.), Proceedings of the 10th International Symposium on Aviation Psychology (pp. 357–361). Columbus: Ohio State University.
40.
Povenmire, H. K., Rockway, M. R., Bunecke, J. L., & Patton, M. W. (1989). Cockpit resource management skills enhance combat mission performance in a B-52 simulator. In R. S. Jensen (Ed.), Proceedings of the 5th International Symposium on Aviation Psychology (pp. 489–494). Columbus: Ohio State University.
41.
Robertson, M. M., & Taylor, J. C. (1995). Team training in aviation maintenance settings: A systematic evaluation. In B. J. Hayward & A. R. Lowe (Eds.), Applied aviation psychology: Achievement, change, and challenge. Proceedings of the Third Australian Aviation Psychology Symposium (pp. 373–383). Aldershot, UK: Avebury Aviation.
42.
Salas, E., Bowers, C. A., & Edens, E. (Eds.). (2001). Improving teamwork in organizations: Applications of resource management training. Hillsdale, NJ: Erlbaum.
43.
Salas, E., Burke, C. S., Bowers, C. A., & Wilson, K. A. (2001). Team training in the skies: Does crew resource management (CRM) training work?Human Factors, 43, 641–674.
44.
Salas, E., & Cannon-Bowers, J. A. (2000). Designing training systems systematically. In E.A. Locke (Ed.), The Blackwell handbook of principles of organizational behavior (pp. 43–59). Malden, MA: Blackwell.
45.
Salas, E., & Cannon-Bowers, C. A. (2001). The science of training: A decade of progress. Annual Review of Psychology, 52, 471–499.
46.
Salas, E., Fowlkes, J. E., Stout, R. J., Milanovich, D. M., & Prince, C. (1999). Does CRM training improve teamwork skills in the cockpit? Two evaluation studies. Human Factors, 41, 326–343.
47.
Salas, E., Prince, C., Bowers, C., Stout, R., Oser, R. L., & Cannon-Bowers, J. A. (1999). A methodology for enhancing crew resource management training. Human Factors, 41, 161–172.
48.
Salas, E., Wilson, K. A., Burke, C. S., & Bowers, C. A. (2002). Myths about crew resource management training. Ergonomics in Design, 10(4), 20–24.
49.
Simmon, D. A. (1997). How to fix CRM. In R. S. Jensen & L. A. Rakovan (Eds.), Proceedings of the 9th International Symposium on Aviation Psychology (pp. 550–553). Columbus: Ohio State University.
50.
Small, S. D. (1998). What participants learn from anesthesia crisis resource management training [Extended Abstract]. Anesthesiology, 89, U153.
51.
Small, S. D., Wuerz, R. C., Simon, R., Shapiro, N., Conn, A., & Setnik, G. (1999). Demonstration of high-fidelity simulation team training for emergency medicine. Academic Emergency Medicine, 6, 312–323.
52.
Spiker, V. A., Tourville, S. J., Bragger, J., Dowdy, D., & Nullmeyer, R. T. (1999). Measuring C-5 crew coordination proficiency in an operational wing. In Proceedings of the Interservice/Industry Training, Simulation and Education Conference [CD-ROM]. Arlington, VA: National Training Systems Association.
53.
Spiker, V. A., Wilson, D. D., & Deen, G. C. (2003). CRM and mission performance during C-130 mission-oriented simulator training. In R. S. Jensen (Ed.), Proceedings of the 12th International Symposium on Aviation Psychology (pp. 1108–1114). Columbus: Ohio State University.
54.
Tannenbaum, S., & Yukl, G. (1992). Training and development in work organizations. In M. Rosenzweig & L. Porter (Eds.), Annual review of psychology (Vol. 43, pp. 399–441), Palo Alto, CA: Annual Reviews.
55.
Taylor, J. C. (1998, August). Evaluating the effectiveness of maintenance resource management (MRM). Paper presented at the 12th International Symposium on Human Factors in Aviation Maintenance, Washington, DC.
56.
Taylor, J. C. (2000, October). A new model for measuring return on investment (ROI) for safety programs in aviation: An example from airline maintenance resource management (MRM). Paper presented at the Advances in Aviation Safety Conference, Daytona Beach, FL.
57.
Taylor, J. C., Robertson, M. M., Peck, R., & Stelly, J. W. (1993). Validating the impact of maintenance CRM training. In R. S. Jensen (Ed.), Proceedings of the 7th International Symposium on Aviation Psychology (pp. 538–542). Columbus: Ohio State University.
58.
Taylor, J. C., & Thomas, R. L. (2003). Written communication practices as impacted by a maintenance resource management training intervention. Journal of Air Transportation, 8, 69–90.
59.
Thompson, J. S., Tourville, S. J., Spiker, V. A., & Nullmeyer, R. T. (1999). Crew resource management and mission performance during MH-53J combat mission training. In Proceedings of the Inter-service/Industry Training, Simulation and Education Conference [CD-ROM]. Arlington, VA: National Training Systems Association.
60.
Wiegmann, D. A., & Shappell, S. A. (2000). Human error and crew resource management failures in naval aviation mishaps: A review of U.S. Naval Safety Center data, 1990–96. Aviation, Space, and Environmental Medicine, 70, 1147–1151.
61.
Wiegmann, D. A., & Shappell, S. A. (2001). Human error analysis of commercial aviation accidents: Application of the human factors analysis and classification system (HFACS). Aviation, Space, and Environmental Medicine, 72, 1006–1016.
