Developmental Evaluation of a Distributed Online Simulation

Abstract

The formative evaluation of developing training systems presents problems at many levels, and development must be based upon training effectiveness information. Acceptance of subjective estimates of training effectiveness requires that the providers reasonably represent the user population and have some relevant expertise. The U.S. Army is developing a distributed simulation using online gaming technology to address conventional and asymmetric warfare, and requires feedback for development. Formative evaluation by military personnel is being used to address interface usability, minimal acceptable levels of fidelity, and potential effectiveness in training. Results from an Army National Guard and an Active Army unit are presented, with data supporting their acceptability as representative evaluators. Their results indicate considerable potential for the system to meet new asymmetric training needs and optimize Soldiers' training time. The effort also demonstrates the difficulty of establishing validity of users in developmental evaluation.

Get full access to this article

View all access options for this article.

References

Army Demographics 1998. Undersecretary of Defense for Personnel and Readiness, obtained July, 2005 from http://www.defenslink.mill/prhome .

Beal

S. A.

Christ

R. E.

(2005). Evaluation of a Game-Based Trainer for Infantry Platoon Leader Decision Making. Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting. Santa Monica, CA: Human Factors & Ergonomics Society.

Boldovici

J. A.

Bessemer

D. W.

Bolton

A. E.

(2002). The Elements of Training Evaluation. Alexandria, VA: U. S. Army Research Institute for the Behavioral and Social Sciences.

Fober

G. W.

Bredthauer

J. L.

Dyer

J. L.

(2001). Computer background of soldiers in FORSCOM units: FY00. (RR-1778, ADA399393) Alexandria, VA: U. S. Army Research Institute for the Behavioral and Social Sciences.

Hays

R. T.

Singer

M. J.

(1989). Simulation fidelity in training system design: Bridging the gap between reality and training. New York, Springer-Verlag.

Knerr

B. W.

Lampton

D. R.

(June, 2005). An Assessment of the Virtual-Integrated MOUT Training System. (Technical Report 1163), Alexandria, VA: U. S. Army Research Institute for the Behavioral and Social Sciences.

Mayo

Singer

M. J.

Kusumoto

(December, 2005). Massively Multi-Player (MMP) Environments for Asymmetric Warfare. Proceedings of the 27th Interservice/Industry Training Systems and Education Conference. Arlington, VA: National Training Systems Association.

National Research Council, (1985). Human Factors Aspects of Simulation. Jones

E. R.

Hennessy

R. T.

Deutsch

, Washington, DC: National Academy Press.

Nielsen

, (1993). Usability Engineering. San Francisco, CA: Morgan Kaufmann.

10.

Singer

M. J.

Kusumoto

(in preparation) Asymmetric Warfare Virtual Training Technology: Disaster & Force Protection Exercises. Paper submitted to the Interservice/Industry Training, Simulation, and Education Conference, 2006.