This paper investigates the active defense guidance problem for flight vehicles in target-pursuer-defender scenario. In the practical scenario, the target flight vehicle with active defense tries to evade the pursuer, and it is always subject to the limitations of incomplete observation information and observation noise. To handle these limitations, this paper proposes a novel cooperative active defense guidance based on a convolutional dueling double deep Q-network (CD3QN) reinforcement learning algorithm. Firstly, considering the spatiotemporal continuity properties of flight vehicles, a stacking mechanism is introduced to transform incomplete observational data into a plane tensor. Based on the mechanism, convolutional neural networks are further employed to effectively extract the feature tensor from the stacked information, which is then utilized by the dueling deep Q-network to derive the guidance law. The CD3QN algorithm resolves the partially observed problem in terms of the correlation between the feature tensor and the state. Moreover, a continuous reward function is shaped based on environmental potential functions, which ensures the optimality invariance and tackles the sparse reward problem during the training process of CD3QN. Finally, numerical experiments are performed to demonstrate the convergence, efficiency, performance and robustness of the proposed active defense guidance.
LiuSYanBZhangT, et al.Three-dimensional coverage-based cooperative guidance law with overload constraints to intercept a hypersonic vehicle. Aero Sci Technol2022; 130: 107908.
2.
ShenZYuJDongX, et al.Penetration trajectory optimization for the hypersonic gliding vehicle encountering two interceptors. Aero Sci Technol2022; 121: 107363.
3.
DingYYueXChenG, et al.Review of control and guidance technology on hypersonic vehicle. Chin J Aeronaut2022; 35(7): 1–18.
4.
TaoCXintaoWSongyanW, et al. Linear-quadratic and norm-bounded differential game combined guidance strategy against active defense aircraft in three-player engagement. Chin J Aeronaut2023; 36(8): 331–350.
5.
LiaoTLiSChenX, et al.Numerical solution of spacecraft pursuit-evasion-capture game based on progressive shooting method. Proc Inst Mech Eng, Part G2023; 237(12): 2874–2886.
6.
LiuFDongXLiQ, et al.Cooperative differential games guidance laws for multiple attackers against an active defense target. Chin J Aeronaut2022; 35(5): 374–389.
7.
WeissMShimaTCastanedaD, et al.Minimum effort intercept and evasion guidance algorithms for active aircraft defense. J Guid Control Dynam2016; 39(10): 2297–2311.
8.
WeissMShimaTCastanedaD, et al.Combined and cooperative minimum-effort guidance algorithms in an active aircraft defense scenario. J Guid Control Dynam2017; 40(5): 1241–1254.
9.
ShalumovV. Optimal cooperative guidance laws in a multiagent target–missile–defender engagement. J Guid Control Dynam2019; 42(9): 1993–2006.
10.
IsaacsR. Differential games: a mathematical theory with applications to warfare and pursuit, control and optimization. Courier Corporation1999.
11.
PerelmanAShimaTRusnakI. Cooperative differential games strategies for active aircraft protection from a homing missile. J Guid Control Dynam2011; 34(3): 761–773.
12.
LiangHWangJWangY, et al.Optimal guidance against active defense ballistic missiles via differential game strategies. Chin J Aeronaut2020; 33(3): 978–989.
13.
LiangHLiZWuJ, et al.Optimal guidance laws for a hypersonic multiplayer pursuit-evasion game based on a differential game strategy. Aerospace2022; 9(2): 97.
14.
DasRJ. Advances in active radar seeker technology. Def Sci J2005; 55: 329–336.
15.
ShafermanVShimaT. Cooperative multiple-model adaptive guidance for an aircraft defending missile. J Guid Control Dynam2010; 33(6): 1801–1813.
16.
HexnerGWeissH. Stochastic approach to optimal guidance with uncertain intercept time. IEEE Trans Aero Electron Syst2010; 46(4): 1804–1820.
17.
BattistiniSShimaT. Differential games missile guidance with bearings-only measurements. IEEE Trans Aero Electron Syst2014; 50(4): 2906–2915.
18.
FonodRShimaT. Estimation enhancement by cooperatively imposing relative intercept angles. J Guid Control Dynam2017; 40(7): 1711–1725.
19.
HuangJZhangHTangG, et al.Robust UKF-Based filtering for tracking a maneuvering hypersonic glide vehicle. Proc Inst Mech Eng, Part G2021; 236(11): 2162–2178.
20.
GaudetBLinaresRFurfaroR. Deep reinforcement learning for six degree-of-freedom planetary landing. Adv Space Res2020; 65(7): 1723–1741.
21.
NiWLiuJLiZ, et al.Cooperative guidance strategy for active spacecraft protection from a homing interceptor via deep reinforcement learning. Mathematics2023; 11(19): 4211.
22.
LiYLiuXHeX, et al.Cooperative optimal guidance of hypersonic glide vehicles by real-time optimization and deep learning. Proc Inst Mech Eng, Part G2023; 237(10): 2266–2283.
23.
YangZLiuXLiuH. Impact time control guidance law with time-varying velocity based on deep reinforcement learning. Aero Sci Technol2023; 142: 108603.
24.
FinnCLevineS. Deep visual foresight for planning robot motion. In: 2017 IEEE international conference on robotics and automation (ICRA). Singapore, 29 May–03 Jun 2017, pp. 2786–2793.
25.
EbertFFinnCDasariS, et al.Visual foresight: model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812005682018.
26.
WangFZhuXZhouZ, et al.Deep-reinforcement-learning-based UAV autonomous navigation and collision avoidance in unknown environments. Chin J Aeronaut2024; 37(3): 237–257.
27.
KongXZhangCXiongF. Impact speed and angle constrained guidance law for unpowered gliding vehicle. Chin J Aeronaut2025; 38(4): 103196–103208.
28.
LiJSongSShiX. Deep reinforcement learning based trajectory real-time planning for hypersonic gliding vehicles. Proc Inst Mech Eng, Part G2024; 238(16): 1665–1682.
29.
WangXYangMWangS, et al.Two-stage game guidance strategy with impact point for active defense aircraft in two-on-two engagement. IEEE Trans Aero Electron Syst2024; 61(1): 710–729.
30.
ChenYChenGPanL, et al.DRQN-based 3D obstacle avoidance with a limited field of view. In: 2021 IEEE/RSJ international conference on intelligent robots and systems (IROS), Prague, Czech Republic, 27 September, 2021, pp. 8137–8143.
DengTHuangHFangY, et al.Reinforcement learning-based missile terminal guidance of maneuvering targets with decoys. Chin J Aeronaut2023; 36(12): 309–324.
33.
GuoYJiangZHuangH, et al.Intelligent maneuver strategy for a hypersonic pursuit-evasion game based on deep reinforcement learning. Aerospace2023; 10(9): 783.
34.
SunGAyepah-MensahDXuR, et al.End-to-end CNN-based dueling deep Q-Network for autonomous cell activation in Cloud-RANs. J Netw Comput Appl2020; 169: 102757.
35.
QiNSunQZhaoJ. Evasion and pursuit guidance law against defended target. Chin J Aeronaut2017; 30(6): 1958–1973.
36.
CandeliATommasiGLuiDG, et al.A deep deterministic policy gradient learning approach to missile autopilot design. IEEE Access2022; 10: 19685–19696.
37.
ShenHZhangKHongM, et al.Towards understanding asynchronous advantage actor-critic: convergence and linear speedup. IEEE Trans Signal Process2023; 71: 2579–2594.
38.
SaarkarAKAnanthasayanamMRSrinivasanT, et al.Comparison of the radar and seeker modes of pursuer guidance. J Guid Control Dynam2009; 32(6): 1912–1920.
