The use of systematic training effectiveness evaluation (TEE) in flightcrew training provides training practitioners with methods to assess the validity and efficacy of specific curricula or training methods for set goals and objectives and deeper insight into the processes and systems used. As discussed in our extended abstract, there are various frameworks used to assess training effectiveness, but there have been few systematic comparisons of these TEE frameworks and little consensus as to which frameworks should be applied for the evaluation of modern training configurations. This poster will overview lessons learned from our comparison of various traditional, factors-based, and systematic frameworks and their respective qualities and limitations. We will provide a series of practical suggestions for the flightcrew training community and outline a baseline of the outcomes and factors to be considered when conducting a TEE in the flightcrew training domain, highlighting future directions for TEE research and practice.
AlvarezK.SalasE.GarofanoC. M. (2004). An integrated model of training evaluation and effectiveness. Human Resource Development Review, 3(4), 385-416. https://doi.org/10.1177/1534484304270820
3.
BatesR. (2004). A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning, 27(3), 341-347. https://doi.org/10.1016/j.evalprogplan.2004.04.011
4.
Cannon-BowersJ. A.SalasE.TannenbaumS. I.MathieuJ. E. (1995). Toward theoretically based principles of training effectiveness: A model and initial empirical investigation. Military Psychology, 7(3), 141-164. https://doi.org/10.1207/s15327876mp0703_1
5.
ChildsJ. M.BellH. B. (2010). Training systems evaluation. In CharltonS. G.O'BrienT. G. (Eds.), Handbook of Human Factors Testing and Evaluation (2nd ed.). CRC Press. https://doi.org/10.1201/9781003000815
6.
DessingerJ. C.MoseleyJ. L. (2006), The full scoop on full-scope evaluation. In PershingJ. A. (ed.), Handbook of Human Performance Technology: Principles, Practices, and Potential (pp. 312-330). San Francisco, CA: Pfeiffer.
HoltonE. F.III (2005). Holton's evaluation model: New evidence and construct elaborations. Advances in Developing Human Resources, 7(1), 37-54. https://doi.org/10.1177/1523422304272080
11.
International Organization for Standardization (ISO). (2019). ISO/IEC 25020:2019(en): Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — Quality measurement framework. ISO Online Browsing Platform (OBP).
12.
KirkpatrickD. L. (1956, May01). How to start an objective evaluation of your training program. Journal of the American Society of Training Directors, 10, 18-22.
13.
KirkpatrickD. (1996). Great ideas revisited: Techniques for evaluating training programs. Training and Development, 50(1), 54–59. (Originally published 1959).
SalasE.TannenbaumS. I.KraigerK.Smith-JentschK. A. (2012). The science of training and development in organizations: What matters in practice. Psychological Science in the Public Interest, 13(2), 74-101. https://doi.org/10.1177/1529100612436661
16.
SitzmannT.WeinhardtJ. M. (2019). Approaching evaluation from a multilevel perspective: A comprehensive analysis of the indicators of training effectiveness. Human Resource Management Review, 29(2), 253-269. https://doi.org/10.1016/j.hrmr.2017.04.001
17.
SonnenfeldN. A.NguyenB.BoesserC. T.JentschF. (2021, May). Modern training practices for flightcrew training of procedural knowledge. International Symposium on Aviation Psychology, 21, 305-309. https://doi.org/10.5399/osu/1148
18.
SottilareR. A.GoldbergB. S.BrawnerK. W.HoldenH. K. (2012). A modular framework to support the authoring and assessment of adaptive computer-based tutoring systems (CBTS). Interservice/Industry Training, Simulation, and Education Conference 2012, 12017. https://www.gifttutoring.org/attachments/download/144/12017.pdf
19.
StufflebeamD. L. (2001a). Evaluation models. New Directions for Evaluation, 2001(89), 7-98. https://doi.org/10.1002/ev.3
