Operationally validating military and defense queuing models requires the rigorous assessment of agreement between functional responses of the model and the system or process of interest. This article provides and contextualizes two distance-based validation methods for operationally validating complex transient-phase military and defense queuing models. The limits of agreement approach and the probability of agreement approach, both developed within the measurement system comparison literature, are contextualized through an illustrative M/M/1 queuing model application derived from the military air traffic control domain. The limits of agreement approach characterizes agreement through an evaluation of the differences between observations and predictions on the same entity, while the probability of agreement approach uses a mixed-effects structural model to characterize the relationship between observations and predictions. The procedures and results of both methods are juxtaposed against common Boolean-based statistical methods and are used to establish global predictive capabilities useful for calibrating military and defense queuing models over multiple settings of controllable input.
MIL-STD-3022. Documentation of verification, validation, and accreditation (VV&A) for models and simulations. Arlington, VA: Department of Defense, 2012.
2.
NikolicN.The 110th anniversary of queueing theory: its applications in the military. Vojnotehnicki Glasnik2019; 67: 806–819.
LedwithMCHillRRChampagneLE, et al. Chapter 14: A review of methods for simulation validation. Handbook of military and defense operations, second edition. Boca Raton, FL: CRC Press, 2025.
5.
LiLLiuFLongG, et al. Performance analysis and optimal allocation of layered defense m/m/n queueing systems. Mathematical problems in engineering. London: Hindawi, 2016, pp. 1–21.
6.
YaoHMaL. The air defense missile weapon system based on queuing theory mixed deployment effectiveness evaluation. In: Proceedings of the 7th international conference on management, education, information and control (MEICI2017), Shenyang, China, 15–17 September 2017, pp. 877–881. Dordrecht: Atlantis Press.
7.
ZhengMWangCWangYJ, et al. Research on aircraft sortie generation rate using multi-class closed queueing network. Appl Mech Mater2013; 380-384: 1864–1867.
8.
JenkinsPRRobbinsMJLundayBJ.Examining military medical evacuation dispatching policies utilizing a Markov decision process model of a controlled queueing system. Ann Oper Res2018; 271: 641–678.
9.
PanticT.Possibility to apply the mass servicing theory method to supersonic aircraft squadron servicing. Vojnotehnicki Glasnik2002; 50: 612–622.
10.
KaczynskiWLeemisLDrewJ.Modeling and analyzing transient military air traffic control. In: Proceedings of the 2010 Winter Simulation Conference, Baltimore, MD, 5–8 December 2010, pp. 1395-1406. New York: IEEE.
BahadoriMMohammadnejhadSMRavangardR, et al. Using queuing theory and simulation model to optimize hospital pharmacy performance. Iran Red Crescent Med J2014; 16: e16807.
13.
DwyerJJ.Analysis of military entry control point queueing. Master’s Thesis, Air Force Institute of Technology, Wright-Patterson Air Force Base, OH, 2016.
14.
PetrovicVMrdakVLukovicB.Mass serving theory application to the analysis of maintenance system functioning. Vojnotehnicki Glasnik2013; 61: 159–181.
15.
GueKRKangK.Staging queues in material handling and transportation systems. In: Proceedings of the 2001 Winter Simulation Conference, Arlington, VA, 9–12 December 2001, pp. 1104–1108. New York: IEEE.
16.
WeinLMCraftDLKaplanEH.Emergency response to an anthrax attack. Proc Natl Acad Sci USA2003; 100: 4346–4351.
17.
McConnellBM.Assessing uncertainty and risk in an expeditionary military logistics network. PhD Thesis, North Carolina State University, Raleigh, NC, 2018.
18.
SargentRG.Verification and validation of simulation models. J Simul2013; 7: 12–24.
19.
PingMFeiyanMMingY.A knowledge-based validation method of complex simulation models. In: 2010 International conference on artificial intelligence and computational intelligence, Sanya, China, 23–24 October 2010, pp. 3–6. New York: IEEE.
20.
SargentRG. An introductory tutorial on verification and validation of simulation models. In: Proceedings of the 2015 winter simulation conference (WSC), Huntington Beach, CA, 06–09 December 2015, pp. 1729–1740. New York: IEEE.
21.
RobinsonS.Chapter verification, validation, and confidence. Simulation: the practice of model development and use. London: Palgrave Macmillan, 2014, pp. 251–268.
22.
SargentRGGoldsmanDMYaacoubT. A tutorial on the operational validation of simulation models. In: Proceedings of the 2016 winter simulation conference (WSC), Washington, DC, 11–14 December 2016, pp. 163–177. New York: IEEE.
23.
RaunakMOlsenM. Quantifying validation of discrete event simulation models. In: Proceedings of the 2014 Winter Simulation Conference, Savannah, GA, 07–10 December 2014, pp. 628–639. New York: IEEE.
24.
SinisiSAlimguzhinVManciniT, et al. Reconciling interoperability with efficient verification and validation within open source simulation environments. Simul Model Pract Theory2021; 109: 102277.
25.
SargentRGGoldsmanDMYaacoubT. Use of the interval statistical procedure for simulation model validation. In: Proceedings of the 2015 winter simulation conference (WSC), Huntington Beach, CA, 06–09 December 2015, pp. 1–13. New York: IEEE.
26.
WackerlyDDMendenhallWScheafferRL.Mathematical statistics with applications. Brooks/Cole: Cengage Learning, 2008.
27.
BalciOSargentRG.Validation of multivariate response models using Hotelling’s two-sample t2 test. Simulation1982; 39: 185–192.
28.
NaylorTHFingerJM.Verification of computer simulation models. Manag Sci1967; 14: B-92–B-101.
29.
AndersonTW.On the distribution of the two-sample Cramer-von Mises criterion. Ann Math Stat1962; 33: 1148–1159.
30.
LawAM.Simulation modeling and analysis. 4th ed. Chicago, IL: McGraw-Hill, 2007.
31.
SargentRG.An interval statistical procedure for use in validation of simulation models. J Simul2015; 9: 232–237.
32.
LimentaniGBRingoMCYeF, et al. Beyond the t-test: statistical equivalence testing. Anal Chem2005; 77: 221A–226A.
33.
LiuYChenWArendtP, et al. Toward a better understanding of model validation metrics. J Mech Des2011; 133: 071005.
34.
RebbaRMahadevanS.Validation of models with multivariate output. Reliab Eng Syst Saf2006; 91: 861–871.
35.
FersonSOberkampfWLGinzburgL.Model validation and predictive capability for the thermal challenge problem. Comput Methods Appl Mech Eng2008; 197: 2408–2430.
36.
OberkampfWLBaroneMF.Measures of agreement between computation and experiment: validation metrics. J Comput Phys2006; 217: 5–36.
37.
AltmanDGBlandJM.Measurement in medicine: the analysis of method comparison studies. Stat1983; 32: 307–317.
38.
BlandJMAltmanDG.Statistical methods for assessing agreement between two methods of clinical measurement. Lancet1986; 327: 307–310.
39.
ChoudharyPKNagarajaHNNagarajaHN, et al. Chapter: Measuring agreement in method comparison studies—a review. Advances in ranking and selection, multiple comparisons, and reliability. Basel, Switzerland: Birkhauser. 2005, pp. 215–244.
40.
LudbrookJ.Confidence in Altman-bland plots: a critical review of the method of differences. Clin Exp Pharmacol Physiol2010; 37: 143–149.
41.
PollockMAKaneJWLomaxK, et al. Method comparison—a different approach. Ann Clin Biochem1992; 29: 556–560.
42.
BlandJMAltmanDG.Applying the right statistics: analyses of measurement studies. Ultrasound Obstet Gynecol2003; 22: 85–93.
43.
StevensNTSteinerSHMacKayRJ.Assessing agreement between two measurement systems: an alternative to the limits of agreement approach. Stat Methods Med Res2017; 26: 2487–2504.
44.
BarnettVD.Simultaneous pairwise linear structural relationships. Biometrics1969; 25: 129–142.
45.
LedwithMCHillRRChampagneLE, et al. Probabilities of agreement for computational model validation. J Verif Valid Uncert2023; 8: 011003.
46.
StevensNTAnderson-CookCM.Comparing the reliability of related populations with the probability of agreement. Technometrics2017; 59: 371–380.
47.
StevensNTRigdonSEAnderson-CookCM.Bayesian probability of agreement for comparing the similarity of response surfaces. J Qual Technol2019; 52: 67–80.
48.
BlandJMAltmanDG.Agreement between methods of measurement with multiple observations per individual. J Biopharm Stat2007; 17: 571–582.
49.
BlandJMAltmanDG.Measuring agreement in method comparison studies. Stat Methods Med Res1999; 8: 135–160.
50.
DewitteKFierensCStöcklD, et al. Application of the Bland–Altman plot for interpretation of method-comparison studies: a critical investigation of its practice. Clin Chem2002; 48: 799–801; authorreply 801.
51.
ManthaSRoizenMFFleisherLA, et al. Comparing methods of clinical measurement: reporting standards for Bland and Altman analysis. Anesth Analg2000; 90: 593–602.
52.
ChhapolaVKanwalSKBrarR.Reporting standards for Bland–Altman agreement analysis in laboratory research: a cross-sectional survey of current practice. Ann Clin Biochem2014; 52: 382–386.
53.
StevensNTAnderson-CookCM.Quantifying similarity in reliability surfaces using the probability of agreement. Qual Eng2017; 29: 395–408.
54.
StevensNTRigdonSEAnderson-CookCM.Bayesian probability of predictive agreement for comparing the outcome of two separate regressions. Qual Reliab Eng Int2018; 34: 968–978.
