Using After-Action Review Based on Automated Performance Assessment to Enhance Training Effectiveness

Abstract

Training simulators have become increasingly popular tools for instructing humans on performance in complex environments. However, the question of how to provide individualized and scenario-specific assessment and feedback to students remains largely an open question. In this work, we follow-up on previous evaluations of the Automated Expert Modeling and Automated Student Evaluation (AEMASE) system, which automatically assesses student performance based on observed examples of good and bad performance in a given domain. The current study provides a rigorous empirical evaluation of the enhanced training effectiveness achievable with this technology. In particular, we found that students given feedback via the AEMASE-based debrief tool performed significantly better than students given only instructor feedback on two out of three domain-specific performance metrics.

Get full access to this article

View all access options for this article.

References

Abbott

R. G.

(2002). Automated tactics modeling: Techniques and applications. Dissertation Abstracts International, 68.

Abbott

R. G.

(2006). Automated expert modeling for automated student evaluation. Intelligent Tutoring Systems, 4053, 1–10.

Anderson

C. W.

Draper

B. A.

Peterson

D. A.

(2000). Behavioral cloning of student pilots with modular neural networks. In ICML ′00: Proceedings of the Seventeenth International Conference on Machine Learning, (pp. 25–32). San Francisco: Morgan Kaufmann.

Corbett

A. T.

(2001). Cognitive computer tutors: Solving the two-sigma problem. In UM ′01: Proceedings of the 8^th International Conference on User Modeling, (pp. 137–147). London: Springer-Verlag.

Jensen

Nolan

Chen

D. Y.

(2005). Automatic causal explanation analysis for combining arms training AAR. In Proceedings of the Industry/Interservice, Training, Simulation and Education Conference (I/ITSEC)..

Murray

(1999). Authoring intelligent tutoring systems: An analysis of the state of the art. International Journal of Artificial Intelligence in Education, 10, 98–129.

Smith-Jentsch

K. A.

Zeisig

R. L.

Acton

McPherson

J. A.

(1998). Team dimensional training: A strategy for guided team self-correction. In Salas

Cannon-Bowers

J. A.

(Eds). Making Decisions under Stress: Implications for Individual and Team Training, (pp. 271–297). Washington, D.C.: APA.

Stevens

S. M.

Forsythe

Abbott

R. G.

Gieseler

C. J.

(2009). Experimental assessment of accuracy of automated knowledge capture. Foundations of Augmented Cognition, Neuroergonomics, and Operational Neuroscience, 5638, 212–216.