Design Specifications for the Scenario-Based Training Automated Collection and Evaluation System (SBT-ACES)

Abstract

Evaluation is a powerful mechanism for realizing engaging, effective and efficient training. The data distilled from summative and formative evaluations can be used to iteratively improve synthetic learning environments, scenario-based content, and assessment tools, thereby enabling more personally meaningful learning experiences. In this paper, we leverage the science of training, psychometric theory, and artificial intelligence to inform the design of the Scenario-Based Training - Automated Collection and Evaluation System (SBT-ACES). SBT-ACES provides instructors with an integrated analytics toolset to facilitate the collection, manipulation, analysis and visualization of evaluative data from a federation of systems, titled the Instructional Support System (ISS). The ISS is a constellation of applications, databases and devices supporting Marine Air Ground Task Force (MAGTF) training via the Digital Virtual Training Environment (DVTE), a laptop-based simulation suite deployed worldwide. A machine learning component within the ISS can leverage post-synthesized SBT-ACES data to continuously tune training such that, over time, accumulated evidence enables content to be adaptively tailored to optimize trainees' learning experiences.

Get full access to this article

View all access options for this article.

References

Aguinis

Kraiger

(in press). Benefits of training and development for individuals and teams, organizations, and society. To appear in Annual Review of Psychology..

Brinkerhoff

R.O.

(1987). Achieving results from training. San Francisco, CA: Jossey-Bass.

Bloom

B.S.

(1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13, 3–16.

Brown

K.G.

Gerhardt

M.W.

(2002). Formative evaluation: An integrative practice model and case study. Personnel Psychology, 55, 951–983.

Camp

Paas

Rikers

van Merrienboer

(2001). Dynamic problem selection in air traffic control training: A comparison between performance, mental effort and mental efficiency. Computers in Human Behavior, 17, 575–595.

Campbell

Kuncel

(2002). Individual and team training. In Anderson

Ones

Sinangil

Viswesvaran

(Eds.), Handbook of Industrial, Work and Organizational Psychology (pp.72–312). London: Sage.

Cannon-Bowers

Burns

Salas

Pruitt

(1998). Advanced technology in scenario-based training. In Cannon-Bowers

Salas

(Eds.), Making decisions under stress, (pp. 365–374). Washington, DC: APA.

Day

Lance

(2004). Understanding the development of leadership complexity through latent growth modeling. In Day

Zaccaro

S.J.

Halpin

S.M.

(Eds.), Leader development for transforming organizations. Mahwah, NJ: Erlbaum.

Ericsson

K.A.

(2009). Enhancing the development of professional performance: Implications from the study of deliberate practice. In Ericsson

K.A.

(Ed.), The development of professional performance: Approaches to objective measurement and designed learning environments. Cambridge University Press.

10.

Fowlkes

Dwyer

D.J.

Oser

R.J.

Salas

(1998). Event-based approach to training (EBAT). The International Journal of Aviation Psychology, 8, 209–221.

11.

Ford

K.J.

Wroten

S.P.

(1984). Introducing new methods for conducting training evaluation and for linking training evaluation to program redesign. Personnel Psychology, 37, 651–665.

12.

IBM (2008). Unlocking the DNA of the adaptable workforce, the IBM Human Capital Study. Milwaukee, WI: IBM.

13.

Kozlowski

S.W.J.

Brown

Weissbein

Cannon-Bowers

J.A.

Salas

(2000). A multilevel perspective on training effectiveness: enhancing horizontal and vertical transfer. In Klein

K.J.

Kozlowski

S.W.J.

(Eds.), Multilevel theory, research, and methods in organizations (pp.157–210). San Francisco: Jossey-Bass.

14.

Kraiger

(2002). Decision-based evaluation. In Kraiger

(Ed.), Creating, implementing, and maintaining effective training and development: Lessons for practice (pp.331–375). San Francisco: Jossey-Bass.

15.

Kraiger

Ford

J.K.

Salas

(1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, 311–328.

16.

Lawshe

(1975). A quantitative approach to content validity. Personnel Psychology, 28, 563–575.

17.

Mirabella

Lamb

J.C.

(1966). Computer based adaptive training applied to symbolic displays. Perceptual and Motor Skills, 23, 647–661.

18.

Paradise

(2007). State of the Industry: ASTD's Annual Review of Trends in Workplace Learning and Performance. Alexandria, VA: ASTD.

19.

Salas

Stagl

K.C.

(2009). Design training systematically by following the science of training. In Locke

E.A.

(Ed.), The Blackwell Handbook of Principles of Organizational Behavior. Blackwell.

20.

Schatz

Bowers

Nicholson

(In press). Advanced situated tutors: Design, philosophy, and a review of existing systems. In Nicholson

Fiore

Vogel-Walcutt

Schatz

(Chairs) , Advancing the science of training in scenario-based training. Symposium presented at HFES 2009, San Antonio, TX.

21.

Stagl

K. C.

Salas

Day

D.V.

(2008). Assessing team learning outcomes: Improving team learning and performance. In Sessa

V.I.

London

(Eds.), Work group learning: Understanding, assessing and improving how groups learn in organizations (pp.369–392). New York: Taylor.