A requirement of an Agent-based Simulation (ABS) is that the agents must be able to adapt to their environment. Many ABSs achieve this adaption through simple threshold equations due to the complexity of incorporating more sophisticated approaches. Threshold equations are when an agent behavior changes because a numeric property of the agent goes above or below a certain threshold value. Threshold equations do not guarantee that the agents will learn what is best for them. Reinforcement learning is an artificial intelligence approach that has been extensively applied to multi-agent systems but there is very little in the literature on its application to ABS. Reinforcement learning has previously been applied to discrete-event simulations with promising results; thus, reinforcement learning is a good candidate for use within an Agent-based Modeling and Simulation (ABMS) environment. This paper uses an established insurgency case study to show some of the consequences of applying reinforcement learning to ABMS, for example, determining whether any actual learning has occurred. The case study was developed using the Repast Simphony software package.
BanksCMSokolowskiJA.Modeling the Niger Delta Insurgency. Soc Sci J2010; 47: 271–293.
2.
ArneyDCArneyK.Modeling insurgency, counter-insurgency, and coalition strategies and operations. J Defense Model Simulat2013; 10: 57–73.
3.
SokolowskiJABanksCMMorrowB.Using an agent-based model to explore troop surge strategy. J Defense Model Simulat2012; 9: 173–186.
4.
NorthMJMacalCM.Managing business complexity: discovering strategic solutions with agent-based modeling and simulation. New York: Oxford University Press, 2007.
5.
SchellingTC.Dynamic models of segregation. J Math Sociol1971; 1: 143–186.
6.
ReynoldsCW.Flocks, herds | schools: a distributed behavioral model. In: proceedings of the 14th annual conference on computer graphics and interactive techniques - SIGGRAPH ‘87, New York, 1987, pp.25–34.
7.
WolframS.A new kind of science. Champaign, IL: Wolfram Media, 2002.
8.
IzquierdoLRPolhillJG.Is your model susceptible to floating-point errors?J Artif Soc Soc Simulat2006; 9: 4.
9.
SuttonRSBartoAG.Reinforcement learning: an introduction. Cambridge, MA: MIT Press, 1998.
10.
LeslieAM.Learning: association or computation? Introduction to the special section. Curr Dir Psychol Sci2001; 10: 124–127.
11.
YoungHP.The possible and the impossible in multi-agent learning. Artif Intell2007; 171: 429–433.
12.
FudenbergDLevineDK.The theory of learning in games. London: MIT Press, 1998.
13.
HudasGVamvoudakisKGMikulskiD. Online adaptive learning for team strategies in multi-agent systems. J Defense Model Simulat2012; 9: 59–69.
14.
ValluriANorthMJMacalCM.Reinforcement learning in supply chains. Int J Neural Syst2009; 19: 331–344.
15.
WangYCUsherJM.Application of reinforcement learning for agent-based production scheduling. Eng Appl Artif Intell2005; 18: 73–82.
16.
WatkinsCJCH. Learning from delayed rewards. Doctoral Thesis, Cambridge University, 1989.
17.
RummeryGA.Problem solving with reinforcement learning. Doctoral Thesis, Cambridge University, 1995.
PiddM.Tools for thinking: modelling in management science. Chichester: Wiley, 1996.
21.
Ali-AkpajiakSPykeT.Measuring poverty in Africa, in Oxford working papers. Oxford: Oxford University, 2003.
22.
SmithDJ.A culture of corruption: everyday deception and popular discontent in Nigeria, Princeton, NJ: Princeton University Press, 2007.
23.
ForestJJSousaMV.Oil and terrorism in the New Gulf: framing US energy and security policy for the Gulf of Guinea. New York: Rowman & Littlefield Publishers, 2006.
24.
FerrariSDaughertyG.A Q-learning approach to automated unmanned air vehicle demining. J Defense Model Simulat2012; 9: 83–92.
25.
StilmanB.Linguistic geometry: from search to construction. Operations Research/Computer Science Interfaces Series. Norwell, MA: Springer, 2000.
26.
PopkenDCoxL.A simulation-optimization approach to air warfare planning. J Defense Model Simulat2004; 1: 127–140.
SuttonRSBartoAG.Reinforcement learning: an introduction. London: MIT Press, 1998.
32.
MichieDChambersRA.BOXES: an experiment in adaptive control. In: DaleEMichieD (eds) Machine intelligence 2. Edinburgh: Oliver and Boyd, 1968, pp.137–152.
33.
CollinsAThomasL.Comparing reinforcement learning approaches for solving game theoretic models: a dynamic airline pricing game example. J Oper Res2012; 63: 1165–1173.
34.
BridleJS. Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters. In: advances in neural information processing systems: proceedings of the 1989 conference, Morgan Kaufmann, San Mateo, USA, 1990, pp.211–217.
NorthMJCollierNTVosJR.Experiences creating three implementations of the repast agent modeling toolkit. ACM Trans Model Comput Simulat2006; 16: 1–25.
37.
BinmoreK.Fun and games: a text on game theory. Lexington, MA: D.C. Heath, 1991.
38.
CollinsAJ.Evaluating reinforcement learning for game theory application: learning to price airline seats under competition. Doctoral Thesis, University of Southampton2009.
39.
GleickJ.Chaos: making a new science. New York: Penguin, 1998.
40.
JungesRKlüglF.Programming agent behavior by learning in simulation models. Appl Artif Intell2012; 26: 349–375.
