A novel actor-critic algorithm is introduced and applied to zero-sum differential game. The proposed novel structure consists of two actors and a critic. Different actors represent the control policies of different players, and the critic is used to approximate the state-action utility function. Instead of neural network, the fuzzy inference system is applied as approximators for the actors and critic so that the specific practical meaning can be represented by the linguistic fuzzy rules. Since the goals of the players in the game are completely opposite, the actors for different players are simultaneously updated in opposite directions during the training. One actor is updated updated toward the direction that can minimize the Q value while the other updated toward the direction that can maximize the Q value. A pursuit-evasion problem with two pursuers and one evader is taken as an example to illustrate the validity of our method. In this problem, the two pursuers the same actor and the symmetry in the problem is used to improve the replay buffer. At the end of this paper, some confrontations between the policies with different training episodes are conducted.
IsaacsR., Differential games: a mathematical theory with applications to warfare and pursuit, control and optimization, 01 1965.
2.
HoY., BrysonA. and BaronS., Differential games and optimal pursuit-evasion strategies, IEEE Transactions on Automatic Control10(4) (1965), 385–389.
3.
LiubarshchukI. and AlthoeferI., The problem of approach in differential–difference games, International Journal of Game Theory45, 02 2015.
4.
MakkapatiV.R., SunW. and TsiotrasP., Optimal evading strategies for two-pursuer/one-evader problems, Journal of Guidance, Control, and Dynamics41(4) (2018), 851–862.
5.
LimS.H., FurukawaT., DissanayakeG. and Durrant-WhyteH.F., A time-optimal control strategy for pursuit-evasion games problems, In, IEEE International Conference on Robotics and Automation, 2004. Proceedings, ICRA ’04. 20044 (2004), 3962–3967.
6.
SunW., TsiotrasP., LollaT., SubramaniD.N. and LermusiauxP.F.J., Pursuit-evasion games in dynamic flow fields via reachability set analysis, In 2017 American Control Conference (ACC), pages 4595–4600, 2017.
7.
WangY., DongL. and SunC., Cooperative control for multi-player pursuit-evasion games with reinforcement learning, Neurocomputing412 (2020), 101–114.
8.
LiangH., WangJ., LiuJ. and LiuP., Guidance strategies for interceptor against active defense spacecraft in two-on-two engagement, Aerospace Science and Technology96 (2020), 105529.
9.
StaddonJ.E.R., The dynamics of behavior: Review of sutton and barto: Reinforcement learning: An introduction (2nd ed.), Journal of the Experimental Analysis of Behavior113(2) (2020), 485–491.
10.
LiuJ., LiuS., WuH. and ZhangY., A pursuit-evasion algorithm based on hierarchical reinforcement learning, Measuring Technology and Mechatronics Automation, International Conference on2 (2009), 482–486.
11.
KiumarsiB., VamvoudakisK.G., ModaresH. and LewisF., Optimal and autonomous control using reinforcement learning: A survey, IEEE Transactions on Neural Networks and Learning SystemsPP (2017), 1–21.
12.
JiaS., WangX. and ShenL., A continuous-time markov decision process-based method with application in a pursuit-evasion example, IEEE Transactions on Systems, Man, and Cybernetics: Systems46 (2015), 1–11.
13.
WangH.-N., LiuN., ZhangY.-Y., FengD.-W., HuangF., LiD.-S. and ZhangY., Deep reinforcement learning: a survey, Frontiers of Information Technology & Electronic Engineering, 10 2020.
14.
MnihV., KavukcuogluK., SilverD., GravesA., AntonoglouI., WierstraD. and RiedmillerM., Playing atari with deep reinforcement learning, 12 2013.
15.
LillicrapT., HuntJ., PritzelA., HeessN., ErezT., TassaY., SilverD. and WierstraD., Continuous control with deep reinforcement learning, CoRR, 09 2015.
16.
WangL., WangM. and YueT., A fuzzy deterministic policy gradient algorithm for pursuit-evasion differential games, Neurocomputing362(Oct.14) (2019), 106–117.
17.
DesoukyS.F. and SchwartzH.M., Self-learning fuzzy logic controllers for pursuit–evasion differential games, Robotics and Autonomous Systems59(1) (2011), 22–33.
18.
AwhedaM. and SchwartzH., A residual gradient fuzzy reinforcement learning algorithm for differential games, International Journal of Fuzzy Systems19 (2017), 1058–1076.
19.
ZhouY., van KampenE.-J. and ChuQ., Hybrid hierarchical reinforcement learning for online guidance and navigation with partial observability, Neurocomputing331 (2019), 443–457.
20.
GrondmanI., BusoniuL., LopesG. and BabuskaR., A survey of actor-critic reinforcement learning: Standard and natural policy gradients, IEEE Transactions on Systems Man and Cybernetics Part B-Cybernetics42 (2012), 1291–1307.
21.
VamvoudakisK.G. and LewisF., Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem, Automatica46 (2010), 878–888.
22.
SilverD., LeverG., HeessN., DegrisT., WierstraD. and RiedmillerM., Deterministic policy gradient algorithms, In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, ICML’14, page I–387–I–395. JMLR.org, 2014.
23.
SongR., WeiQ. and SongB., Neural-network-based synchronous iteration learning method for multi-player zero-sum games, Neurocomputing242 (2017), 73–82.
24.
JouffeL., Fuzzy inference system learning by reinforcement methods, Trans Sys Man Cyber Part C28(3) (1998), 338–355.
25.
MnihV., KavukcuogluK., SilverD., RusuA., VenessJ., BellemareM., GravesA., RiedmillerM., FidjelandA., OstrovskiG., PetersenS., BeattieC., SadikA., AntonoglouI., KingH., KumaranD., WierstraD., LeggS. and HassabisD., Human-level control through deep reinforcement learning, Nature518 (2015), 529–533.
26.
RiedmillerM., Neural fitted q iteration - first experiences with a data efficient neural reinforcement learning method, Mach. Learn.: ECML, page 317–328, 2005.
27.
WatkinsJ.C.H. and DayanP., Q-learning, Mach Learn8 (1992), 279–292.
28.
GuS., LillicrapT., SutskeverI. and LevineS., Continuous deep q-learning with model-based acceleration, 03 2016.
29.
WangZ., SchaulT., HesselM., HasseltHado V., LanctotM. and FreitasN.D., Dueling network architectures for deep reinforcement learning, In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, page 1995–2003. JMLR.org, 2016.
30.
SuttonR., McallesterD., SinghS. and MansourY., Policy gradient methods for reinforcement learning with function approximation, Adv Neural Inf Process Syst12 (2000), 1057–1063.
31.
ProkhorovD.V. and WunschD., Adaptive critic design, IEEE transactions on neural networks / a publication of the IEEE Neural Networks Council8 (1997), 997–1007.
32.
TAKAGITOMOHIRO and SUGENOMICHIO, Fuzzy identification of systems and its applications to modeling and control, IEEE Trans. Systems, Man, and Cybernet15 (1985), 116–132.
33.
BravoL., RuizU. and Murrieta-CidRafael, A pursuit–evasion game between two identical differential drive robots, Journal of the Franklin Institute357(10) (2020), 5773–5808.
34.
AkametaluA., GhoshS., FisacJ. and TomlinC., A minimum discounted reward hamilton-jacobi formulation for computing reachable sets, 09 2018.
