Computer simulations have revolutionized the analysis of military scenarios. As computing power has advanced, simulations can now incorporate intricate tactical-level engagements. However, accurately representing actors’ decisions at this level poses new challenges for developing and validating these simulations. In this context, this paper presents the methodologies and lessons learned from a study conducted to assess the application of agent-based modeling and simulation (ABMS) in analyzing beyond visual range (BVR) air combat scenarios, focusing on the influence of agent behavior on the outcomes. The proposed approach integrates real pilots into a face validation phase to examine symmetric and asymmetric engagements. The results underscore the significance of agent behaviors for the outcomes, for example, showing how specific behaviors are capable of mitigating the advantages of superior weaponry. Furthermore, the research explores the dynamics of aircraft acting in pairs, demonstrating the potential to evaluate tactics and the impact of numerical advantage. Ultimately, the results enhance the simulations’ credibility and confirm their plausibility, in line with the face validation methodology. This powerful phase bolsters subsequent steps in the overall validation process. In addition, the findings show how specific configurations of the agents, including tactical coordination, can significantly affect the simulation outcomes and validity.
HillRRMillerJO. A history of United States military simulation. In: Proceedings of the 2017 winter simulation conference (WSC), Las Vegas, NV, 3–6 December 2017, pp. 346–364. New York: IEEE.
2.
HillRRTolkAHodsonDD, et al. Open challenges in building combat simulation systems to support test, analysis and training. In: Proceedings of the 2018 winter simulation conference, Gothenburg, 9–12 December 2018, pp. 3730–3741. New York: IEEE.
3.
Office of the Assistant Secretary of Defense for Acquisition Washington United States. Unmanned systems integrated roadmap 2017-2042. Fort Belvoir, VA, 1August2018, https://apps.dtic.mil/sti/citations/AD1059546 (accessed 24 March 2022).
4.
MacalCM. Everything you need to know about agent-based modelling and simulation. J Simul2016; 10: 144–156.
5.
WeimerCWMillerJOHillRR. Agent-based modeling: an introduction and primer. In: Proceedings of the 2016 winter simulation conference (WSC), Washington, DC, 11–14 December 2016, pp. 65–79. New York: IEEE.
6.
FosterKLPettyMD. Estimating the tactical impact of robot swarms using a semi-automated forces system and design of experiments methods. J Def Model Simul2021; 18: 247–269.
7.
PadgettNE. Defensive swarm: an agent-based modeling analysis. Master’s Thesis, Naval Postgraduate School, https://calhoun.nps.edu/handle/10945/56777 (2017, accessed 28 June 2021).
8.
Andrew AuTHoekPJLoEHS. Combat analysis of joint force options using agent-based simulation. In: Proceedings of the 2018 military communications and information systems conference (MilCIS 2018), Canberra, ACT, Australia, 13–15 November 2018. New York: IEEE.
9.
WilliamsRA. Lessons learned on development and application of agent-based models of complex dynamical systems. Simul Model Pract Th2018; 83: 201–212.
10.
ToubmanA. Validating air combat behaviour models for adaptive training of teams. In: RSottilareJSchwarz (eds) Adaptive instructional systems, vol. 11597 (Lecture notes in computer science; including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics). Cham: Springer, 2019, pp. 557–571.
11.
LeeJ-SFilatovaTLigmann-ZielinskaA, et al. The complexities of agent-based modeling output analysis. JASSS: J Artif Soc S. Epub ahead of print 31October2015. DOI: 10.18564/jasss.2897.
12.
LeeJShinSParkM, et al. Agent-based simulation and Its application to analyze combat effectiveness in network-centric warfare considering communication failure environments. Math Probl Eng2018; 2018: 2730671.
13.
ChristensenCSalmonJ. An agent-based modeling approach for simulating the impact of small unmanned aircraft systems on future battlefields. J Def Model Simul2020; 19: 481–500.
14.
McCourtRNgKMitchellR. An agent—based approach towards network—enabled capabilities—I: simulation validation and illustrative examples. J Def Model Simul2013; 10: 313–326.
15.
ConnorsCDMillerJOLundayBJ. Using agent-based modeling and a designed experiment to simulate and analyze a new air-to-air missile. J Def Model Simul2016; 13: 321–330.
16.
De Lima FilhoGMKuroswiskiARMedeirosFLL, et al. Optimization of unmanned air vehicle tactical formation in war games. IEEE Access2022; 10: 21727–21741.
17.
DantasJPACostaANGeraldoD, et al. Engagement decision support for beyond visual range air combat. In: Proceedings of the 2021 Latin American robotics symposium (LARS), 2021 Brazilian symposium on robotics (SBR), and 2021 workshop on robotics in education (WRE), Natal, Brazil, 11–15 October 2021, pp. 96–101. New York: IEEE.
18.
PoropudasJVirtanenK. Game-theoretic validation and analysis of air combat simulation models. IEEE T Syst Man Cy A2010; 40: 1057–1070.
19.
HuangCDongKHuangH, et al. Autonomous air combat maneuver decision using Bayesian inference and moving horizon optimization. J Syst Eng Electron2018; 29: 86–97.
20.
ToubmanA. Calculated moves: generating air combat behaviour. Doctoral Thesis, Leiden University, https://hdl.handle.net/1887/84692 (2020, accessed 27 April 2022).
21.
TurcaníkM. Modeling CGF behavior with machine learning techniques: requirements and future directions; ATC-remote control tower view project smart bandits-machine learning view project. In: Proceedings of the interservice/industry training, simulation, and education conference, Orlando, FL, 30 November–3 December 2015, pp. 1–11. Amsterdam: National Aerospace Laboratory, NLR.
22.
MacedoH. Alocação Coordenada de Múltiplos Alvos Aéreos Utilizando Heurística Especializada. Master’s Thesis, Instituto Tecnológico de Aeronáutica (ITA), 2017, http://www.bdita.bibl.ita.br/tesesdigitais/
23.
YaoJHuangQWangW. Adaptive human behavior modeling for air combat simulation. In: Proceedings of the 2015 IEEE/ACM 19th international symposium on distributed simulation and real time applications (DS-RT), Chengdu, China, 14–16 October 2015. New York: IEEE.
24.
FloydMWKarneebJMooreP, et al. A goal reasoning agent for controlling UAVs in beyond-visual-range air combat. In: Proceedings of the 26th international joint conference on artificial intelligence (IJCAI), Melbourne, VIC, Australia, 19–25 August 2017, pp. 4714–4721. Washington, DC: Navy Center for Applied Research in AI, Naval Research Laboratory.
25.
YangZZhouDPiaoH, et al. Evasive maneuver strategy for UCAV in beyond-visual-range air combat based on hierarchical multi-objective evolutionary algorithm. IEEE Access2020; 8: 46605–46623.
26.
ErnestNCarrollDSchumacherC, et al. Genetic fuzzy based artificial intelligence for unmanned combat aerial vehicle control in simulated air combat missions. J Def Manag2016; 6: 1000144.
LawAM. How to build valid and credible simulation models. In: Proceedings of the 2009 winter simulation conference (WSC), Austin, TX, 13–16 December 2009, pp. 24–33. New York: IEEE.
OtaDPradhanM. Modular verification and validation for NATO generic vehicle architecture-based land platforms. In: Proceedings of the 2018 international conference on military communications and information systems (ICMCIS), Warsaw, 22–23 May 2018, pp. 1–7. New York: IEEE.
31.
LeeK. A framework for verification and validation of multi resolution modeling federation using live, virtual, and constructive simulations. PhD Thesis, University of Central Florida, https://stars.library.ucf.edu/etd2020/722 (2021, accessed 3 March 2022).
32.
ApplegetJBlaisCJayeM. Best practices for US Department of Defense model validation: lessons learned from irregular warfare models. J Def Model Simul2013; 10: 395–410.
33.
DantasJPACostaANGomesVCF, et al. ASA: a simulation environment for evaluating military operational scenarios. In: Proceedings of the world congress in computer science, computer engineering, and applied computing (CSCE), Las Vegas, NV, 25–28 July 2022.
34.
Van OijenJToubmanA. Are we machine learning yet? Computer generated forces with learning capabilities in military simulation. In: Proceedings of the interservice industry, training, simulation, and education conference (I/ITSEC), 2021, https://www.xcdsystem.com/iitsec/proceedings/index.cfm?Year=2021&AbID=96958&CID=862#View (2021, accessed 17 January 2022).
BerndtJS. JSBSim: an open source flight dynamics model in C++. In: Proceedings of the AIAA modeling and simulation technologies conference—collection of technical papers, Providence, RI, 16–19 August 2004, vol. 1, pp. 261–287. Reston, VA: AIAA.
37.
LiWShiJWuY, et al. A multi-UCAV cooperative occupation method based on weapon engagement zones for beyond-visual-range air combat. Def Technol2022; 18: 1006–1022.
38.
RobersonD. The unified behavior framework for the simulation of autonomous agents. Master’s Thesis, Air Force Institute of Technology, Dayton, OH, 2015.
39.
KamrudAJHodsonDDPetersonGL, et al. Unified behavior framework in discrete event simulation systems. J Def Model Simul2017; 14: 471–481.
40.
SargentRG. Verification and validation of simulation models. J Simul2013; 7: 12–24.
41.
LorscheidIHeineBOMeyerM. Opening the “black box” of simulations: increased transparency and effective communication through the systematic design of experiments. Comput Math Organ Th2012; 18: 22–62.
42.
SchervishMJLawAMKeltonWD. Simulation modeling and analysis. J Am Stat Assoc1983; 78: 743.
43.
ShawRL. Fighter combat: tactics and maneuvering. Annapolis, MD: Naval Institute Press, 1985.
