In this paper, a formation cooperative tracking control problem for unmanned aerial vehicles is investigated via non–zero-sum game theory and reinforcement learning. The formation cooperative tracking control problem is converted to a multi-player non–zero-sum game problem to attain the team optimality of multiple unmanned aerial vehicles. The desired trajectory can be achieved via a game-based feedback controller to minimize the global cost function involving all control policies. The Nash equilibrium control laws are learned based on only the collected system data, without knowledge of the vehicle dynamics. The asymptotic convergence of the constructed global closed-loop control system is proven via theoretical analysis. Simulation results validate the effectiveness of the proposed control method.
ChengMLiuHWenG, et al. (2023) Data-driven time-varying formation-containment control for a heterogeneous air-ground vehicle team subject to active leaders and switching topologies. Automatica153: 111029.
2.
FisacJFAkametaluAKZeilingerMN, et al. (2018) A general safety framework for learning-based control in uncertain robotic systems. IEEE Transactions on Automatic Control64(7): 2737–2752.
3.
GaoWJiangZPLewisFL, et al. (2018) Leader-to-formation stability of multiagent systems: An adaptive optimal control approach. IEEE Transactions on Automatic Control63(10): 3581–3587.
4.
GuD (2008) A differential game approach to formation control. IEEE Transactions on Control Systems Technology16(1): 85–93.
5.
HuoYWangDQiaoJ, et al. (2022) Off-policy model-free learning for multi-player non-zero-sum games with constrained inputs. IEEE Transactions on Circuits and Systems I: Regular Papers70(2): 910–920.
6.
JiangYJiangZP (2012) Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics. Automatica48(10): 2699–2704.
7.
LewisFLVrabieDLSyrmosVL (2012) Optimal Control. Hoboken, NJ: John Wiley.
8.
LiWLiQLiSE, et al. (2022) Indirect shared control through non-zero sum differential game for cooperative automated driving. IEEE Transactions on Intelligent Transportation Systems23(9): 15980–15992.
9.
LinW (2014) Distributed UAV formation control using differential game approach. Aerospace Science and Technology35: 54–62.
10.
LinXLiuJYuY, et al. (2020) Event-triggered reinforcement learning control for the quadrotor UAV with actuator saturation. Neurocomputing415: 135–145.
11.
LiuHLiDKimJ, et al. (2014) Real-time implementation of decoupled controllers for multirotor aircrafts. Journal of Intelligent & Robotic Systems73: 197–207.
12.
LiuYAWangQDongC, et al. (2019) Time-varying formation control for unmanned aerial vehicles with external disturbances. Transactions of the Institute of Measurement and Control41(13): 3777–3786.
13.
LuJWeiQWangZ, et al. (2022) Event-triggered optimal control for discrete-time multi-player non-zero-sum games using parallel control. Information Sciences584: 519–535.
14.
MaHJXuLXYangGH (2021) Multiple environment integral reinforcement learning-based fault-tolerant control for affine nonlinear systems. IEEE Transactions on Cybernetics51(4): 1913–1928.
15.
MuslimovTZMunasypovRA (2020) Adaptive decentralized flocking control of multi-UAV circular formations based on vector fields and backstepping. ISA Transactions107: 143–159.
16.
OdekunleAGaoWDavariM, et al. (2020) Reinforcement learning and non-zero-sum game output regulation for multi-player linear uncertain systems. Automatica112: 108672.
17.
SongRLewisFLWeiQ (2016) Off-policy integral reinforcement learning method to solve nonlinear continuous-time multiplayer nonzero-sum games. IEEE Transactions on Neural Networks and Learning Systems28(3): 704–713.
18.
SuHZhangHSunS, et al. (2020) Integral reinforcement learning-based online adaptive event-triggered control for non-zero-sum games of partially unknown nonlinear systems. Neurocomputing377: 243–255.
19.
VamvoudakisKGLewisFLHudasGR (2012) Multi-agent differential graphical games: Online adaptive learning solution for synchronization with optimality. Automatica48(8): 1598–1611.
20.
WeiXYangJFanX (2020) Fully distributed guidance laws for unmanned aerial vehicles formation flight. Transactions of the Institute of Measurement and Control42(5): 965–980.
21.
XuLXWangYLWangX, et al. (2023) Distributed active disturbance rejection formation tracking control for quadrotor UAVs. IEEE Transactions on Cybernetics. Epub ahead of print 1November2023. DOI: 10.1109/TCYB.2023.3324752.
22.
YanDZhangWChenH, et al. (2023) Robust control strategy for multi-UAVs system using MPC combined with Kalman-consensus filter and disturbance observer. ISA Transactions135: 35–51.
23.
YinTGuZXieX (2022) Observer-based event-triggered sliding mode control for secure formation tracking of multi-UAV systems. IEEE Transactions on Network Science and Engineering10(2): 887–898.
24.
ZhangJYanJZhangP (2020a) Multi-UAV formation control based on a novel back-stepping approach. IEEE Transactions on Vehicular Technology69(3): 2437–2448.
25.
ZhangJZhangPYanJ (2020b) Distributed adaptive finite-time compensation control for UAV swarm with uncertain disturbances. IEEE Transactions on Circuits and Systems I: Regular Papers68(2): 829–841.
26.
ZhaoWLiuHLewisFL (2021) Data-driven fault-tolerant control for attitude synchronization of nonlinear quadrotors. IEEE Transactions on Automatic Control66(11): 5584–5591.
27.
ZhaoWLiuHLewisFL (2020) Robust formation control for cooperative underactuated quadrotors via reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems32(10): 4577–4587.
