Using a mock simulation event and Monte Carlo simulation to compare alternative network architectures for distributed training simulation

Abstract

A systems engineering effort produced three alternative network architectures for a system to support simulation-based training events. A mock simulation event was used to compare the effort required for each alternative to prepare for, conduct, and analyze an event. The mock event was a manual execution of a process for conducting simulation events consisting of a sequence of defined activities. As a mock event, no simulations were executed and the activities’ work was not performed. Instead, the labor required for each activity for each alternative was estimated by simulation subject matter experts using the Delphi method. The experts’ estimates drove a set of Monte Carlo trials. Probability distributions were parameterized with the estimates, from which values for the labor required for each activity for each alternative were stochastically generated. Those values were summed to estimate the mean total labor required for an event for each alternative. Confidence intervals for each alternative’s labor did not overlap, and hypothesis tests confirmed that the alternatives’ labor requirements were statistically different. The methodology, which combined a mock simulation event, Delphi estimation, Monte Carlo simulation, and statistical analysis, is likely to be useful in similar situations where simulation system architecture alternatives are compared before implementation.