Multi-UAV formation control via deep reinforcement learning and multi-step experience storage in dense obstacle environments

Abstract

This paper presents a deep reinforcement learning (DRL)-based Multi-Agent Control for Formation and Obstacle Avoidance (MACFOA) algorithm to solve collaborative formation and obstacle avoidance decision-making for unmanned aerial vehicle (UAV) systems in dense obstacle environments. The algorithm primarily addresses the coupled strategy update challenges that emerge from simultaneous obstacle avoidance and formation control. A distributed control framework is employed to enable efficient formation and obstacle avoidance for UAV swarms in densely obstructed environments. To address the inefficiency and instability of strategy update problems due to sparse data samples in DRL formation control algorithms, an enhanced multi-step continuous experience replay mechanism is introduced. This mechanism stores and leverages experience data from multiple consecutive time steps, linking contextual information while fully accounting for temporal dependencies, ensuring continuous dynamic policy optimization throughout the training process. Comparative simulations were carried out to evaluate the performances of the proposed approach in terms of efficiency and flexibility. The results have shown that employing the MACFOA-MULT4 algorithm, which utilizes a four-step experience replay strategy, leads to optimal performance, significantly enhancing both training efficiency and stability. Compared to MACFOA, it reduces root mean square error (RMSE) by 38.56% and improves by 25.17% over the multi-agent recurrent deterministic policy gradient (MADRPG). In dynamic simulations on the AirSim platform, the algorithm demonstrated strong adaptability and stability, especially in high-obstacle-density environments. Its superior performance in control stability and task efficiency highlights the effectiveness and advantages of the proposed control strategy.