Games and simulators can be a valuable platform to execute complex multi-agent, multiplayer, imperfect information scenarios with significant parallels to military applications: multiple participants manage resources and make decisions that command assets to secure specific areas of a map or neutralize opposing forces. These characteristics have attracted the artificial intelligence (AI) community by supporting development of algorithms with complex benchmarks and the capability to rapidly iterate over new ideas. The success of AI algorithms in real-time strategy games such as StarCraft II has also attracted the attention of the military research community aiming to explore similar techniques in military counterpart scenarios. Aiming to bridge the connection between games and military applications, this work discusses past and current efforts on how games and simulators, together with the AI algorithms, have been adapted to simulate certain aspects of military missions and how they might impact the future battlefield. This paper also investigates how advances in virtual reality and visual augmentation systems open new possibilities in human interfaces with gaming platforms and their military parallels.
CaffreyMBJr. On wargaming. Newport, Rhode Island: U.S. Naval War College Press.
2.
PerlaPP. The art of wargaming: a guide for professionals and hobbyists. Annapolis, Maryland: Naval Institute Press, 1990.
3.
VinyalsOEwaldsTBartunovS, et al. StarCraft II: a new challenge for reinforcement learning. arXiv preprint 1708.04782, 2017.
4.
SunPSunXHanL, et al. TStarBots: defeating the cheating level builtin AI in StarCraft II in the full game. Corr2018; abs/180907193.
5.
VinyalsOBabuschkinICzarneckiWM, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature2019; 575: 350–354.
6.
WaytowichNBartonSLLawhernV, et al. Grounding natural language commands to StarCraft II game states for narration-guided reinforcement learning. In: Artificial intelligence and machine learning for multi-domain operations applications, vol. 11006. International Society for Optics and Photonics, p.110060S.
7.
WaytowichNBartonSLStumpE, et al. A narration-based reward shaping approach using grounded natural language commands. In: International conference on machine learning (ICML), workshop on imitation, intent and interaction, 2019.
8.
WangXSongJQiP, et al. SCC: an efficient deep reinforcement learning agent mastering the game of StarCraft II. arXiv preprint Arxiv:201213169, 2020.
9.
HanLXiongJSunP, et al. TStarBot-X: an open-sourced and comprehensive study for efficient league training in StarCraft II full game. arXiv preprint Arxiv:201113729, 2020.
10.
BernerCBrockmanGChanB, et al. Dota 2 with large scale deep reinforcement learning. Corr2019; abs/19120 6680.
11.
MnihVKavukcuogluKSilverD, et al. Playing Atari with deep reinforcement learning. arXiv preprint Arxiv:13125602, 2013.
12.
MnihVKavukcuogluKSilverD, et al. Human-level control through deep reinforcement learning. Nature2015; 518: 529–533.
13.
HesselMModayilJvan HasseltH, et al. Rainbow: combining improvements in deep reinforcement learning. Proc AAAI Conf Artif Intell2018; 32: 3215–3222.
14.
SilverDHuangAMaddisonCJ, et al. Mastering the game of Go with deep neural networks and tree search. Nature2016; 529: 484–489.
15.
SilverDSchrittwieserJSimonyanK, et al. Mastering the game of Go without human knowledge. Nature2017; 550: 354–359.
16.
SchrittwieserJAntonoglouIHubertT, et al. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature2020; 588: 604–609.
17.
CampbellMHoaneAJJrHsuFh. Deep blue. Artif Intell2002; 134: 57–83.
18.
HsuFH. Behind deep blue: building the computer that defeated the world chess champion. Princeton, NJ: Princeton University Press, 2002.
19.
BrownNSandholmT. Superhuman AI for heads-up no-limit poker: libratus beats top professionals. Science2018; 359: 418–424.
20.
GortneyWE. Department of defense dictionary of military and associated terms. Technical report, Joint Chiefs of Staff, Washington, DC, 2010.
21.
BurnsSDella VolpeDBabbR, et al. War gamers handbook: a guide for professional war gamers. Technical report, US Naval War College, Newport, RI, 2015.
22.
ArmyU. Army doctrine publication no. 3-0: operations. Washington, DC: Headquarters, Department of the Army, 2019.
23.
KottA. Advanced technology concepts for command and control. Bloomington, IN: Xlibris Corporation, 2004.
24.
KottAGroundLBuddR, et al. Toward practical knowledge-based tools for battle planning and scheduling. In: The Eighteenth national conference on artificial intelligence (AAAI-02) was held July 28 – August 1, 2002, at the Shaw conference center in Edmonton, Alberta, Canada pp. 894–899. Menlo Park, CA: American Association for Artificial Intelligence.
25.
OwnbyMKottA. Reading the mind of the enemy: predictive analysis and command effectiveness. Technical report, Solers Inc., Arlington, VA, 2006.
26.
KottACorpacPS. COMPOEX technology to assist leaders in planning and executing campaigns in complex operational environments. Technical report, Defense Advanced Research Projects Agency, Arlington, VA, 2007.
27.
KottALudwigJLangeM. Assessing mission impact of cyberattacks: toward a model-driven paradigm. IEEE Sec Priv2017; 15: 65–74.
28.
WalshMMentheLGeistE, et al. Exploring the feasibility and utility of machine learning-assisted command and control: volume 1, findings and recommendations. Santa Monica, CA: RAND Corporation, 2021.
29.
WalshMMentheLGeistE, et al. Exploring the feasibility and utility of machine learning-assisted command and control: volume 2, supporting technical analysis. Santa Monica, CA: RAND Corporation, 2021.
SchulmanJMoritzPLevineS, et al. High-dimensional continuous control using generalized advantage estimation. arXiv preprint Arxiv:150602438, 2015.
32.
BoronJDarkenC. Developing combat behavior through reinforcement learning in wargames and simulations. In: 2020 IEEE conference on games (CoG), Osaka, Japan, 24–27 August 2020, pp. 728–731. New York: IEEE.
33.
LanchesterFW. Aircraft in warfare: the dawn of the fourth arm. London: Constable Limited, 1916.
CorpsUM. Marine Corps doctrinal publication 1-0. Washington, DC: Marine Corps Operations, 2011.
37.
AsherDEZaroukianEBartonSL. Adapting the predator-prey game theoretic environment to Army tactical edge scenarios with computational multiagent systems. arXiv preprint Arxiv:180705806, 2018.
38.
AsherDGarber-BarronMRodriguezS, et al. Multi-agent coordination profiles through state space perturbations. In: 2019 international conference on computational science and computational intelligence (CSCI), Las Vegas, NV, 5–7 December 2019, pp. 249–252. New York: IEEE.
39.
ZaroukianERodriguezSSBartonSL, et al. Algorithmically identifying strategies in multi-agent game-theoretic environments. In: Artificial intelligence and machine learning for multi-domain operations applications, 14–18 April 2019, Baltimore, Maryland, United States, vol. 11006, p. 1100614. Bellingham, WA: SPIE.
40.
AsherDEZaroukianEPerelmanB, et al. Multi-agent collaboration with ergodic spatial distributions. In: Artificial intelligence and machine learning for multi-domain operations applications II, 27 April–9 May 2020, Online Only, California, United States, vol. 11413. Bellingham, WA: SPIE, p. 114131N.
41.
BartonSLWaytowichNRAsherDE. Coordination-driven learning in multi-agent problem spaces. arXiv preprint Arxiv:180904918, 2018.
42.
BartonSLAsherD. Reinforcement learning framework for collaborative agents interacting with soldiers in dynamic military contexts. In: Next-generation analyst VI, 15–19 April 2018, Orlando, Florida, United States, vol. 10653, pp. 106503. Bellingham, WA: SPIE.
43.
FuQFanCLSongY, et al. Alpha C2 —an intelligent air defense commander independent of human decision-making. IEEE Access2020; 8: 87504–87516.
SchwartzPJO’NeillDVBentzME, et al. AI-enabled wargaming in the military decision making process. In: Artificial intelligence and machine learning for multi-domain operations applications II, vol. 11413, 27 April–9 May 2020, Online Only, California, United States, p. 114130H. Bellingham, WA: SPIE.
47.
ReesePP. Military decisionmaking process: lessons and best practices. Technical report, Center for Army Lessons Learned, Fort Leavenworth, KS, 2015.
48.
EspeholtLSoyerHMunosR, et al. IMPALA: scalable distributed deep-RL with importance weighted actor-learner architectures. In: International conference on machine learning, 10–15 July 2018, Stockholmsmässan, Stockholm Sweden, pp. 1407–1416. PMLR.
49.
KapturowskiSOstrovskiGDabneyW, et al. Recurrent experience replay in distributed reinforcement learning. In: International conference on learning representations. New Orleans, Louisiana, United States, 6–9 May 2019. ICLR.
50.
LiangELiawRNishiharaR, et al. RLlib: abstractions for distributed reinforcement learning. In: DyJKrauseA (eds) Proceedings of the 35th international conference on machine learning, proceedings of machine learning research, vol. 80. 10–15 July 2018, Stockholmsmässan, Stockholm Sweden, PMLR, pp. 3053–3062.
51.
NarayananPVindiolaMParkS, et al. First-year report of ARL director’s strategic initiative (FY20-23): Artificial Intelligence (AI) for Command and Control (C2) of Multi Domain Operations. Technical Report ARL-TR-9192. Adelphi, MD: Adelphi Laboratory Center, DEVCOM Army Research Laboratory (US), 2021.
52.
MnihVBadiaAPMirzaM, et al. Asynchronous methods for deep reinforcement learning. In: Volume 48: International conference on machine learning, 20–22 June 2016, New York, USA, pp. 1928–1937. PMLR.
53.
HochreiterSSchmidhuberJ. Long short-term memory. Neur Comput1997; 9: 1735–1780.
54.
SurduJRHainesGDPoochUW. OpSim: a purpose-built distributed simulation for the mission operational environment. Simul Ser1999; 31: 69–74.
55.
WittmanRLJrHarrisonCT. OneSAF: a product line approach to simulation development. Technical report, MITRE Corp., Orlando, FL, 2001.
56.
ParsonsDSurduJJordanB. OneSAF: a next generation simulation modeling the contemporary operating environment. In: Proceedings of Euro-simulation interoperability workshop. 27–29 June 2005, Toulouse, France, Citeseer.
57.
BlaisCL. Marine air ground task force (MAGTF) tactical warfare simulation (MTWS). In: Proceedings of winter simulation conference, Lake Buena Vista, FL, 11–14 December 1994, pp. 839–844. New York: IEEE.
58.
BrockmanGCheungVPetterssonL, et al. OpenAI Gym. arXiv preprint Arxiv:160601540, 2016.
59.
AmodeiDOlahCSteinhardtJ, et al. Concrete problems in AI safety. arXiv preprint Arxiv:160606565, 2016.
60.
DennisonMSTroutTT. The accelerated user reasoning for operations, research, and analysis (AURORA) cross-reality common operating picture. Technical Report, Combat Capabilities Development Command, Adelphi, MD, 2020.
