A Human Systems Integration Method for Validating Team Performance Assessment within a Simulation-Based Training System

Abstract

Improving simulation-based training is the focus of a Navy program of research titled Debriefing Distributed Scenario-Based Exercises (DDSBE). DDSBE is intended to optimize training effectiveness by demonstrating improved methods and technologies for assessing team and multi-team performance, diagnosis routines, and debriefing templates. A main objective of the DDSBE program in spiral 1 was to test the training simulator capability to produce valid and reliable team performance assessments. This paper describes a human systems integration test methodology and an exploratory experiment that applied the methodology. The results showed excellent assessment validity and inter-rater reliability when a handheld PC was used to assess team task and teamwork behaviors in conjunction with an automated performance recording capability. The results of this analysis should provide guidance to simulation developers in designing distributed training that supports learning combat team skills.

Get full access to this article

View all access options for this article.

References

Carolan

T. F.

Bilazarian

(2004). Automated data collection and assessment for debriefing distributed simulation based exercises. Proceedings of the Human Factors and Ergonomics Society 48th Annual Meeting [CD-ROM, p. 2567–2571]. Santa Monica, CA.

Hart

S. G.

Stavelend

L. E.

(1988). Development of NASA-TLX (Task Load Index): Results of experimental and theoretical research. In Hancock

P. A.

Meshkati

, Human Mental Workload, (pp. 139–183), Amsterdam: North Holland.

Johnston

J. H.

Radtke

P. H.

Van Duyne

Stretton

Freeman

Bilazarian

(2004) Team training in distributed simulation based exercises. Proceedings of the Human Factors and Ergonomics Society 48th Annual Meeting [CD-ROM, p. 2557–2561]. Santa Monica, CA.

Neville

Fowlkes

Milham

Merket

D. C.

Bergondy

M. L.

Walwanis

Strini

(2001). Training team integration in a large, distributed, tactical team: A cognitive approach. Proceedings of the 23rd Annual Interservice/Industry Training, Simulation, and Education Conference [CD-ROM, p. 1035–1044]. NDIA, Arlington, VA.

Smith-Jentsch

K. A.

Johnston

J. H.

Payne

S. C.

(1998). Measuring team-related expertise in complex environments. In Cannon-Bowers

J. A.

Salas

Making Decisions Under Stress: Implications for Individual and Team Training (pp. 61–87). Washington, DC: American Psychological Association.

Smith-Jentsch

K. A.

Zeisig

R. L.

Acton

McPherson

J. A.

(1998). Team Dimensional Training: A strategy for guided team self-correction. In Cannon-Bowers

J. A.

Salas

Making Decisions Under Stress: Implications for Individual and Team Training (pp. 271–297). Washington, DC: American Psychological Association.

Stretton

M. L.

Wilson

M. S.

(2004). Semi-automated observation and assessment-trainer interaction within a distributed training environment. Proceedings of the Human Factors and Ergonomics Society 48th Annual Meeting [CD-ROM, p. 2572–2576]. Santa Monica, CA.