As knowledge and data increase in scale and complexity, it is more difficult to apply these two key assets to achieve optimal effectiveness in engagement simulation. The aim of this study was to investigate the techniques of knowledge and data integration with respect to the development of smart agents to predict accurate behaviors in tactical engagements. To reduce the complexity of combat behavior representation, with respect to the functions, we represented subject matter expert operational knowledge by proposing multiple levels of cascaded hierarchical structure, namely, the function decision tree, to increase the readability and maintainability of the behavioral model. For decision points in a behavioral model, smart agents can be trained based on data samples collected from rounds of constructive simulations which provide validated physical models and tactical principles. As a proof of concept, we constructed a simulation testbed of multi-warhead ballistic missile penetration, which generated 129,600 constructive simulations over a total of 84 h. Thereafter, we selected 5817 data samples (i.e. ~4.5% of the simulations) using an operational metric of total rewards exceeding 100. The data samples are used to train an artificial neural network and then this network is used to develop a deep reinforcement learning agent. The results revealed that the training process iterated nearly 17,000 epochs until the policy loss decreased to an acceptable low value. The smart agent increased the ratio of ballistic missile target hits by 18.96%, a significant increase when compared with the traditional rule-based behavioral model.
CaoL. A new age of AI: features and futures. IEEE Intell Syst2022; 37: 25–37.
2.
GillSSXuMXOttavianiC, et al. AI for next generation computing: emerging trends and future directions. Internet Things J2022; 19: 1–34.
3.
AbdelmegidMAO’SullivanMGonzálezVA, et al. A case study on the use of a conceptual modeling framework for construction simulation. Simulation2022; 98: 433–460.
4.
SeoKMChoiCKimTG, et al. DEVS-based combat modeling for engagement-level simulation. Simulation2014; 90: 759–781.
5.
McGrawRMMacDonaldRA. Abstract modeling for engineering and engagement level simulations. In: Proceedings of the 2000 winter simulation conference, Orlando, FL, 10–13 December 2000, pp. 326–334. New York: IEEE.
6.
ErnestNCarrollDSchumacherC, et al. Genetic fuzzy based artificial intelligence for unmanned combat aerial vehicle control in simulated air combat missions. J Def Manag2016; 06: 1–7.
7.
PopeAPIdeJSMićovićD, et al. Hierarchical reinforcement learning for air combat at DARPA’s AlphaDogfight Trials. IEEE Trans Artif Intell2022; 1: 1–15.
8.
KimBSKangBGChoiSH, et al. Data modeling versus simulation modeling in the big data era: case study of a greenhouse control system. Simulation2017; 93: 579–594.
9.
BocciarelliPD’AmbrogioAFalconeA, et al. A model-driven approach to enable the simulation of complex systems on distributed architectures. Simulation2019; 95: 1185–1211.
10.
FineganDPZhuJFengXN, et al. The application of data-driven methods and physics-based learning for improving battery safety. Joule2021; 5: 316–329.
11.
NgMFZhaoJYanQY, et al. Predicting the state of charge and health of batteries using data-driven machine learning. Nat Mach Intell2020; 2: 161–170.
12.
IEEE Computer Society. 1516-2010 —IEEE standard for modeling and simulation (M&S) high level architecture (HLA)—framework and rules, http://ieeexplore.ieee.org/servlet/opac?punumber=5553438 (2010, accessed 21 November 2022).
13.
HeQZhangMGongJ. An introduction of BOM modeling framework. IJMLC2011; 1: 353–358.
PetersonJL. Petri net theory and the modeling of systems. 1st ed. Upper Saddle River, NJ: Prentice Hall, 1981.
16.
GillA. Introduction to the theory of finite-state machines. Math Comput1962; 92: 63–74.
17.
SchaadR. Parallel functional decision trees for situated agent control. Dr. Dobb’s J1999; 24: 62–70.
18.
AzarMC. Assessing the treatment of airborne tactical high energy lasers in combat simulations. Master’s Dissertation, Air Force Institute of Technology, Dayton, OH, 2003.
19.
MillerJOJasonLHonabargerB. Modeling and measuring network centric warfare (NCW) with the system effectiveness analysis simulation (SEAS). In: Proceedings of the 11th ICCRTS coalition command and control in the networked era, Dayton, OH, March2006, pp. 1–20.
20.
ChenYZZhangP. Modeling and simulation oriented to the multi-military mission of U.S. army. J Comm Control2018; 4: 89–94.
21.
HuangJZhaoXHaoJG, et al. Brief introduction of KD-HLA: an integrated environment to support M&S based on HLA. In: Proceedings of the second international conference on computer modeling & simulation, Sanya, China, 22–24 January 2010, pp. 281–283. New York: IEEE.
BrandsteinAHorneG. Data farming: a meta-technique for research in the 21st century (Maneuver Warfare Science). Technical report, Naval War College, Newport, RI, 1998.
24.
SanchezSM. Data farming: methods for the present, opportunities for the future. ACM Trans Model Comput Simul2020; 30: 1–30.
25.
FeldkampNBergmannSStrassburgerS. Knowledge discovery in simulation data. ACM Trans Model Comput Simul2020; 30: 1–25.
26.
BarryPZhangJPMcDonaldMM. Architecting a knowledge discovery engine for military commander utilizing massive runs of simulations. In: Proceedings of the ninth ACM SIGKDD international conference on knowledge discovery and data mining, Washington, DC, August2003, pp. 699–704.
27.
FisherRA. The use of multiple measurements in taxonomic problems. Ann Eugen1936; 7: 179–188.
28.
QuinlanJR. Introduction of decision trees. Mach Learn1986; 1: 81–106.
29.
PinedaFJ. Generalization of back-propagation to recurrent neural networks. Phys Rev Lett1987; 59: 2229–2232.
30.
BurgesCJC. A tutorial on support vector machines for pattern recognition. ACM T Intel Syst Tec1998; 2: 121–167.
31.
LiD. A tutorial survey of architectures, algorithms and applications for deep learning. APSIPA Signal Inf Process2014; 3: 1–29.
32.
Paternina-ArboledaCDDasTK. A multi-agent reinforcement learning approach to obtaining dynamic control policies for stochastic lot scheduling problem. Simul Model Pract Theory2005; 13: 389–406.
33.
KesslerCCapocchiLSantucciJF, et al. Hierarchical Markov decision process based on DEVS formalism. In: Proceedings of 2017 winter simulation conference, Las Vegas, NV, 3–6 December 2017, pp. 1001–1012. New York: IEEE.
34.
MnihVKavukcuogluKSilverD, et al. Human-level control through deep reinforcement learning. Nature2015; 518: 529–533.
35.
LillicrapTPHuntJJPritzelA, et al. Continuous control with deep reinforcement learning. In: Proceedings of 4th international conference on learning representations, San Juan, Puerto Rico, May 2016, pp. 1–14.
36.
HasseltVNGuezASilverD. Deep reinforcement learning with double Q-learning. In: Proceedings of the thirtieth AAAI conference on artificial intelligence, Phoenix, AZ, 2016, pp. 2094–2100.
37.
MnihVBadiaAPMirzaM, et al. Asynchronous methods for deep reinforcement learning. In: Proceedings of the 33rd international conference on machine learning, New York, 2016, pp. 1928–1937.
38.
SchulmanJLevineSMoritzP, et al. Trust region policy optimization. In: Proceedings of the 32nd international conference on machine learning, Lille, France, 2015, pp. 1889–1897.
39.
HarrnojaTZhouAAbbeelP, et al. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: Proceedings of the 35th international conference on machine learning, Stockholm, 2018, pp. 1861–1870.
TolkA. The next generation of modeling & simulation: integrating big data and deep learning. In: Proceedings of the conference on summer computer simulation, San Diego, CA, July 2015, pp. 1–8.
42.
AframAShariF. Review of modeling methods for HVAC systems. Appl Therm Eng2014; 67: 507–519.
43.
ThierryASBastienPVittoriE, et al. “Smart entity”-how to build DEVS models from large amount of data and small amount of knowledge. In: Proceedings of simulation tools and techniques, Chengdu, China, August2019, pp. 615–626.
44.
BaeKHMustafeeNLazarova-MolnarS, et al. Hybrid modeling of collaborative freight transportation planning using agent-based simulation, auction-based mechanisms, and optimization. Simulation2022; 98: 753–771.
45.
CristeaAOkamotoT. Knowledge computing method for enhancing the effectiveness of a WWW distance education system. In: BrusilovskyPStockOStrapparavaC (eds) Adaptive hypermedia and adaptive web-based systems. Berlin: Springer, 2000, pp. 2–5.
46.
RoutrayAOsuriKKPattanayakS, et al. Introduction to data assimilation techniques and ensemble Kalman filter. In: MohantyUCGopalakrishnanSG (eds) Advanced numerical modeling and data assimilation techniques for tropical cyclone prediction. Berlin: Springer, 2016, pp. 307–330.
47.
XieX. Data assimilation in discrete event simulations. PhD Thesis, Delft University of Technology, Delft, 2018.
48.
ZhuZLeiYL. Model & data hybrid driven smart modeling for combat systems. In: Proceedings of the 2020 summer simulation conference, San Diego, CA, November 2020, pp. 1–12.
49.
LaroqueCSkooghAGopalakrishnanM. Functional interaction of simulation and data analytics-potentials and existing use-cases. In: Proceedings of simulation in produktion und logistik, Kassel, 2017, pp. 403–412.
50.
SaadawiHWainerGPliegoG. DEVS execution acceleration with machine learning. In: Proceedings of the symposium on theory of modeling & simulation, Pasadena, CA, April 2016, pp. 1–6.
51.
DeistTPattiAWangZ, et al. Simulation assisted machine learning. Bioinformatics2018; 35: 1–11.
52.
ZhangHXHeBSLuGY, et al. A simulation and machine learning based optimization method for integrated pedestrian facilities planning and staff assignment problem in the multi-mode rail transit transfer station. Simul Model Pract Theory2022; 115: 102–449.
53.
ShaoGShinSJJainS. Data analytics using simulation for smart manufacturing. In: Proceedings of the 2014 winter simulation conference, Savannah, GA, December 2014, pp. 2192–2203.
54.
GrievesMW. Product lifecycle management: the new paradigm for enterprises. Int J Prod Dev2005; 2: 71–84.
55.
GlaessgenEStargelD. The digital twin paradigm for future NASA and U.S. Air Force vehicles. In: Proceedings of the 53rd AIAA/ASME/ASCE/AHS/ASC structures, structural dynamics and materials conference, Honolulu, HI, April 2012, pp. 1818–1832.
56.
ZhuZLeiYLSarjoughianH, et al. UML-based combat effectiveness simulation system modeling within MDE. J Syst Eng Electron2018; 29: 1180–1196.
57.
SchaddMSternheimAMBlankendaalR, et al. How a machine can understand the command intent. JDMS. Epub ahead of print 5August2022.
58.
StrembeckMZdunU. An approach for the systematic development of domain-specific languages. Software Pract Exper2010; 39: 1253–1292.
59.
GoodfellowIBengioYCourvilleA. Deep learning. Cambridge, MA: MIT press, 2016.
60.
CohenJCohenPWestSG, et al. Applied multiple regression/correlation analysis for the behavioral sciences. London: Routledge, 2013.
61.
LiuCS. Ornstein–Uhlenbeck process, Cauchy process, and Ornstein–Uhlenbeck–Cauchy process on a circle. Appl Math Lett2013; 26: 957–962.
62.
HyndmanRJKoehlerAB. Another look at measures of forecast accuracy. Int J Forecast2006; 22: 679–688.
63.
KingmaDPBaJL. Adam: a method for stochastic optimization. In: Proceedings of the 3rd international conference on learning representations, San Diego, CA, May 2015, pp. 1–15.
64.
GarrettRKFairbanksJPLoperML, et al. The application of applied category theory to quantify mission success. Simulation2023; 99: 201–220.
