Although instructional research on simulation has been around for almost 40 years, validation research has failed to hold itself to a common, scientifically acceptable methodology for evaluating this type of learning environment. Several comprehensive reviews of simulation assessment literature have all concluded that this problem stems from poorly designed studies, a failure to adhere to a generally accepted research taxonomy, and no well-defined constructs with which to assess learning outcomes. This article seeks to address the problem by reviewing the various concepts employed in the literature of simulation evaluation and integrating them into a coherent framework from which the evaluative process may proceed in a more systematic manner.
Alessi, S. M.
(1988). Fidelity in the design of instructional simulations. Journal of Computer Based Instruction, 15(2), 40-47.
2.
Anderson, P.
, Cannon, H. M., Malik, D.,& Thavikulwat, P. (1998). Games as instruments of assessment. In J. K. Butler Jr., N. H. Leonard,& S. W. Morgan (Eds.), Developments in business simulation and experiential learning(Vol. 25, pp. 31-39). Association for Business Simulation and Experiential Learning/ Savant Electronic Publishing, Inc. Available from http://www.absel.org
3.
Babbie, E.
(1992). The practice of social research(6th ed.). Belmont, CA: Wadsworth.
4.
Blaiwes, A. S.
,& Regan, J. J. (1986). Training devices: Concepts and progress. In J. A. Ellis (Ed.), Military contributions to instructional technology. New York: Praeger.
5.
Bloom, B. S.
, Englehart, N. D., Furst, E. J., Hill, W., & Krathwohl, D. R. (1956). Taxonomy of educational objectives—The classification of educational goals. Handbook I: Cognitive domain. New York: David McKay.
6.
Boocock, S. S.
(1972). Validity-testing of an intergenerational relations game. Simulation & Games, 3, 29-40.
7.
Bredemeier, M. E.
, & Greenblat, C. S. (1981). The educational effectiveness of simulation games. Simulation & Games, 12, 307-332.
8.
Bunker, W. M.
(1978). Training effectiveness versus simulation realism. In Eleventh NTEX/Industry Conference Proceedings. Orlando, FL: Naval Training Equipment Center.
9.
Butler, R. J.
, Markulis, P. M.,& Strang, D. R. (1988). Where are we? An analysis of the methods and focus of the research on simulation gaming. Simulation & Games, 19, 3-26.
10.
Campbell, D. T.
, & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research on teaching. In N. L. Gage (Ed.), Handbook of research and on teaching. Chicago: Rand McNally.
11.
Cannon, H. M.
(1995). Dealing with the complexity paradox in business simulation games. Developments in Business Simulation & Experiential Exercises, 22, 97-103.
12.
Cannon, H. M.
,& Burns, A. C. (1999). A framework for assessing the competencies reflected in simulation performance. Developments in Business Simulation & Experiential Exercises, 26, 40-44.
13.
Carmines, E. G.
, & Zeller, R. A. (1979). Reliability and validity assessment.Beverly Hills, CA: Sage.
14.
Cook, T. D.
, & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton Mifflin.
15.
Cox, J. A.
, Wood, R., Boren, L., & Thorne, H. W. (1965). Functional and appearance fidelity of training devices for fixed procedures tasks(HumRRO Tech. Rep. No. 65-4). Alexandria, VA: Human Resources Research Organization.
16.
Dickson, J. R.
, Whitely, R. R., & Faria, A. J. (1990). An empirical investigation of the internal validity of a marketing simulation game. Developments in Business Simulations& Experiential Exercises, 17, 47-52.
17.
Dukes, R. L.
,& Waller, S. J. (1976). Toward a general evaluation model for simulation games: GEM. Simulation & Games, 7, 75-96.
18.
Dwyer, F. M.
(1974). Strategies for improving visual learning.University Park: Pennsylvania State University, Learning Services.
19.
Gagne, R. M.
(1954). Training devices and simulators: Some research issues. American Psychologist, 9, 95-107.
20.
Greenlaw, P. S.
, Herron, L. W., & Rawdon, R. H. (1962). Business simulation in industrial and university education. Englewood Cliffs, NJ: Prentice Hall.
21.
Greenlaw, P. S.
,& Wyman, F. P. (1973). The teaching effectiveness of games in collegiate business courses. Simulation & Games, 4, 259-294.
22.
Hays, R. T.
,& Singer, M. J. (1989). Simulation fidelity in training system design: Bridging the gap between reality and training. New York: Springer-Verlag.
23.
House, W. C.
, & Napier, H. S. (1988). A comparative study of strategic performance factors in actual and simulated business environments. Developments in Business Simulation & Experiential Exercises, 15, 50-55.
24.
Kibbee, J. M.
(1961). Model building for management games. In A. Newgarden (Ed.), Simulation and gaming: A symposium(pp. 8-15). New York: American Management Association.
25.
Kibler, R. S.
, Cegala, D. J., Barker, L. L., & Miles, D. T. (1974). Objectives for instruction and evaluation. Boston: Allyn & Bacon.
26.
Kinkade, R.
, & Wheaton, G. (1972). Training devices design. In H. Vancoff & R. Kinkade (Eds.), Human engineering guide to equipment design. Washington, DC: American Institutes for Research.
27.
Krathwohl, D. R.
, Bloom, B. S., & Masia, B. B. (1964). Taxonomy of educational objectives—The classification of educational goals. Handbook II: Affective domain. New York: David McKay.
28.
Loehlin, J. C.
(1998). Latent variable models: An introduction to factor, path, and structural analysis(3rd ed.). Mahwah, NJ: Lawrence Erlbaum.
29.
Martin, E. L.
, & Waag, W. L. (1978). Contributions of platform motion to simulator training effectiveness: Study I—Basic contact. Brooks Air Force Base, TX: Air Force Human Resources Laboratory.
30.
Mehrez, A.
, Reichel, A., & Olami, R. (1987). The business game versus reality: A case study in the Israeli environment. Simulation & Games, 18, 488-500.
31.
Mihram, G. A.
(1972). Some practical aspects of the verification and validation of simulation models. Operations Research, 23, 17-29.
32.
Miles, W. G.
, Biggs, W. D.,& Schubert, J. N. (1986). Student perceptions of skill acquisition through cases and a general management simulation: A comparison. Simulation & Games, 17, 7-24.
33.
Miller, G. E.
(1974). Some considerations in the design and utilization of simulators for technical training (Rep. No. AFHRL-TR-74-65). Alexandria, VA: Air Force Human Resource Laboratory.
34.
Miller, J. G.
(1978). Living systems. New York: McGraw-Hill.
35.
Muckler, F. A.
, Nygaard, J. E., O’Kelly, L. I., & Williams, A. C. (1959). Physiological variables in the design of flight simulators for training. Wright-Patterson Air Force Base, OH: Wright Air Development Center.
36.
Norris, D. R.
(1986). External validity of business games. Simulation & Games, 17, 447-459.
37.
Norris, D. R.
, & Snyder, C. A. (1982). External validation of simulation games. Simulation & Games, 13, 73-85.
38.
Parasuraman, A.
(1981). Assessing the worth of business simulation games. Simulation & Games, 12, 189-200.
39.
Pegden, C. D.
, Shannon, R. E.,& Sadowski, R. P. (1995). Introduction to simulation using SIMAN(2nd ed.). Hightstown, NJ: McGraw-Hill.
40.
Pierfy, D.
(1977). Comparative simulation game research: Stumbling bocks and stepping stones. Simulation & Games, 8, 255-268.
41.
Simpson, E. J.
(1974). The classification of educational objectives in the psychomotor domain. In R. S. Kibler, D. J. Cegala, L. L. Barker,& D. T. Miles (Eds.), Objectives for instruction and evaluation. Boston: Allyn & Bacon.
42.
Stanislaw, H.
(1986). Tests of computer simulation validity: What do they measure?Simulation & Games, 17, 173-191.
43.
Teach, R. D.
(1987). Profits: The false prophet. Developments in Business Simulation & Experiential Exercises, 14, 205-207.
44.
Vandierendock, A.
(1975). Inferential simulation: Hypothesis-testing by computer simulation. Nederlands Tijdschrift voor de Psychologie, 30, 677-700.
45.
Wolfe, J.
(1981). Research on the learning effectiveness of business simulation games: A review of the state of the science. Developments in Business Simulation & Experiential Exercises, 8, 72-72.
46.
Wolfe, J.
(1985). The teaching effectiveness of games in collegiate business courses: A 1973-1983 update. Simulation & Games, 16, 251-288.
47.
Wolfe, J.
(1990). The guide to experiential learning. In J. W. Gentry (Ed.), The evaluation of computer-based business games: Methodology, findings, and future needs. New York: Nichols.
48.
Wolfe, J.
, & Crookall, D. (1998). Developing a scientific knowledge of simulation/gaming. Simulation & Gaming, 29, 7-19.
49.
Wolfe, J.
,& Jackson, R. (1989). An investigation of the need for algorithmic validity. Simulation& Games, 20, 272-291.
50.
Wolfe, J.
,& Roberts, C. R. (1986). The external validity of a business management game: Afive-year longitudinal study. Simulation & Games, 17, 45-49.
51.
Wolfe, J.
,& Roberts, C. R. (1993). A further study of the external validity of business games: Five year peer group indicators. Simulation & Gaming, 24, 21-33.
