Adaptive collision-free control for UAVs with discrete-time system based on reinforcement learning

Abstract

A flexible reinforcement learning (RL) optimal collision-avoidance control formulation for unmanned aerial vehicles (UAVs) with discrete-time frameworks is revealed in this work. By utilizing the neural network (NN) estimating capacity and the actor-critic control scheme of the RL technique, an adaptive RL optimal collision-free controller with a minimal learning parameter (MLP) is formulated, which is based on a novel strategic utility function. The optimal collision-avoidance control issue, which couldn’t be addressed in the prior literature, can be resolved by the suggested approaches. Furthermore, the proposed MPL adaptive optimal control formulation allows for a reduction in the quantity of adaptive laws, leading to reduced computational complexity. Additionally, a rigorous stability analysis is provided, demonstrating that the uniform ultimate boundedness (UUB) of all signals in the closed-loop system is ensured by the proposed adaptive RL. Finally, the simulation outcomes illustrate the effectiveness of the proposed optimal RL control approaches.

Introduction

The unmanned aerial vehicles (UAVs) compact and lightweight design, coupled with their capacity to perform tasks inconvenient or challenging for humans, has garnered widespread acclaim. They have demonstrated their prowess in diverse domains, including industrial inspections, emergency response and disaster relief, and daily assistance. Nonetheless, throughout this developmental trajectory, there has been an incremental rise in incidents where quadcopters have inflicted harm upon individuals and property, thereby jeopardizing airspace safety.^1,2 Thus, ensuring the possession of autonomous obstacle avoidance capability in quadcopters is regarded as the paramount and indispensable functional prerequisite for the execution of intricate operational tasks.

Regarding the obstacle avoidance issue in UAVs, a substantial body of literature has already presented various solutions from different perspectives. The obstacle avoidance problem in UAVs primarily refers to the design of a method that enables the UAV to safely reach the target point from the starting point within a specified flight environment. This entails not only considering how to safely navigate the UAVs around obstacles but also satisfying the requirements of the corresponding flight trajectory and the physical constraints of the UAVs itself. ³ The obstacle avoidance method based on the route planning algorithm is also known as the global planning obstacle avoidance algorithm. Its basic idea is to use the route planning algorithm to find a flight path that allows the unmanned aerial vehicle to depart from the starting point, avoid all obstacles, and reach the target point. Then, the route tracking and guidance method is applied to control the unmanned aerial vehicle to fly along the generated route, achieving the purpose of obstacle avoidance.^4–6 The obstacle avoidance method based on the local collision prevention algorithm is also known as local planning obstacle avoidance. It refers to using the local collision prevention controller of UAVs to perform real-time evasion of detected obstacles. These methods do not rely on global information and do not require knowledge of the initial and target points. They only rely on real-time obstacle information detected by UAVs sensors and are commonly used for obstacles with insufficient prior information or sudden obstacles. ⁷ The obstacle avoidance scheme deriving from the artificial potential function method utilizes virtual potential fields to produce attractive and repulsive forces on the UAVs, and forms the resultant force through attraction and repulsion, establishing the low-level regulation controller, thereby obtaining an effective local obstacle avoidance flight path. ⁸ However, traditional UAVs obstacle avoidance algorithms typically require the construction of offline three-dimensional maps. Based on the global map, these algorithms use obstacle points as constraints and employ path planning algorithms to compute the optimal path. While some obstacle avoidance algorithms avoid the complex map construction process, they often require manual adjustment of numerous parameters, and the robots cannot utilize obstacle avoidance experience for self-iteration during the obstacle avoidance process. Based on the aforementioned analysis, there is an urgent demand for enabling intelligent obstacle avoidance in quadcopter unmanned aerial vehicles through real-time feedback and autonomous decision-making in complex environments.

With the advancement of machine learning, in the face of challenges related to integrating the human “trial-and-error-improvement” autonomous learning mechanism into obstacle avoidance control of quadcopter drones, several a priori research propositions are expounded as follows. Researchers have incorporated supervised learning into UAVs obstacle avoidance, treating obstacle avoidance as a classification problem based on supervised learning. ⁹ Reinforcement learning primarily optimizes its own behavior through interaction with the external environment. Its advantage lies in its independence from the offline maps required by traditional non-machine learning methods and the annotated datasets needed for supervised learning. By learning the mapping relationship between input data and output actions through deep learning models, reinforcement learning enables intelligent agents to handle decision-making problems in high-dimensional continuous spaces, avoiding the complex offline map construction work. ¹⁰ A deep reinforcement learning approach, based on uncertain perception, is proposed in which enables quadcopter drones to maintain “vigilance” in unfamiliar unknown environments by estimating collision probabilities. This approach reduces the speed of operation and minimizes the possibility of collisions. ¹¹ The DDPG algorithm is applied to plan the desired path for quadcopter drones and combined it with a PID controller to achieve collision-free target tracking tasks using a hierarchical structure. ¹² The DDPG algorithm, as a classical algorithm for continuous action control, has been widely used in obstacle avoidance, path planning, and other problems. Ding et al. ¹³ divides the path planning task of UAVs into a Path Travel Policy Module and an Information Exploration Policy Module using the DDPG algorithm. To assist in guiding the generation of UAV flight path trajectories and enhancing the model’s learning capabilities, an improved Artificial Potential Field (APF) force-guiding mechanism is introduced in the Path Travel Policy Module. The Information Exploration Policy Module provides the UAV with a series of temporary target points, enabling the UAV to exhibit better obstacle avoidance performance on complex maps. Li et al., ¹⁴ building upon the Proximal Policy Optimization (PPO) algorithm for proximal policy optimization, improves the reward function design by introducing density rewards, distance rewards, and step-length penalties. This reduces congestion occurrences and enhances the task efficiency of the agent. Han et al. ¹⁵ proposes an obstacle avoidance algorithm that combines Artificial Potential Fields with deep reinforcement learning. By modifying the Artificial Potential Field (APF), obstacles directly affect intermediate target positions rather than control commands, making it useful for guiding previously trained one-dimensional deep reinforcement learning controllers. Yan et al. ¹⁶ presents a distributed formation and obstacle avoidance approach based on Multi-Agent Reinforcement Learning (MARL). Agents in the system make decisions and distributed controls using only local and relevant information. In the event of any disconnection, they rapidly reconfigure themselves into a new topology. This method shows improved performance in terms of formation error, formation convergence rate, and obstacle avoidance success rate. Considering the heterogeneity of agents cannot be overlooked in crowded scenarios, Zhu et al. ¹⁷ models agents using Oriented Bounding Capsules (OBC) and transforms the interaction state space of robot-obstacle agent pairs. To address speed heterogeneity, a collision risk function related to speed is designed to shape robot behavior. This method enhances the success rate of collision avoidance for agents in congested scenes. However, it suffers from the problem of overestimation bias in Q-values. When this cumulative error reaches a certain level, it can lead to suboptimal policy updates and divergence behavior.

An aspect of practical significance that warrants attention is the presence of unknown information. In the work by Doukhi and Lee, ¹⁸ a robust adaptive NN control strategy is developed for a quadrotor UAVs. This controller takes into account uncertainties arising from disturbances, inertia, mass, and aerodynamics. By employing adaptive neural networks and certainty equivalent control, the controller approximates the unknown dynamics, obviating the need for precise models or disturbance information. Wang et al. ¹⁹ propose an innovative adaptive control scheme catering to uncertain discrete-time nonlinear systems that presents a rigorous strict-feedback form. This algorithm utilizes a singular neural network approximation to convert the initial system into a predictor form, effectively addressing the noncausal issue. The designed controller is consisted of just one pair of single actor controller and one adaptive compensator, thus streamlining the implementation process and alleviating computational load. To enhance control performance by leveraging estimated unknown information, the evaluation of control performance often involves the utilization of a long-term performance index, which has garnered considerable attention in the literature. In addressing this user-specified long-term cost, the control community frequently employs the reinforcement learning (RL) technique, as highlighted in the works of Refs.^20–22 In Chen et al., ²³ a dynamic surface control scheme using RBFNN and disturbance observer is proposed to handle uncertainty and saturation, avoiding complexity explosion and ensuring convergence of closed-loop signals. In addition, Moreover, in the context of time-varying-constrained nonlinear systems, previous studies^24,27 have utilized barrier Lyapunov functions to guarantee constraint adherence, utilizing NNs for approximating the unknown information in control strategy.

Taking into consideration the aforementioned content, addressing the issue of autonomous intelligent obstacle avoidance for quadcopter drones based on real-time environmental feedback, this paper summarizes the innovations of the “trial-and-error correction” evaluation-action intelligent control framework using reinforcement learning as follows:

1. The article presents an innovative approach that combines neural networks and actor-critic control mechanism of RL to develop an optimal collision-free RL strategy for UAVs with discrete-time systems.

2. The article introduces the idea of utilizing an MLP architectural strategy to reduce the number of adaptive adjustments within the adaptive control framework of RL. This leads to decreased computational complexity and improved operational efficiency.

3. The established RL framework and the MLP-based RL control system guarantee the stability of the closed-loop systems. This is supported by the analysis of uniformly ultimately bounded (UUB) behavior. Furthermore, it leverages the potential of deep learning models to adeptly tackle decision-making obstacles across vast continuous domains.

As for the structure of this work, it is stated as follows: Section “Problem formulation” introduces a nonlinear model of an UAVs and transforms the model into a discrete form. In Section “Design of reinforcement learning control strategy,” a pioneering online reinforcement learning control scheme for UAVs, considering collision risk, is presented. Section “Simulation” showcases simulations that validate the proficiency of the devised controllers. Lastly, Section “Conclusion” offers a conclusion, summarizing the key findings of the paper.

Problem formulation

Six DOF system model

The schematic diagram of the quadrotor UAVs under consideration is taken into consideration. To facilitate the description of motion, two reference frames are introduced: the inertial coordinate system denoted as $O-\mathrm{xyz}$ and the body frame represented as $O_b-x_b{y}_b{z}_b$ . Therefore, the 3-DOF rotational dynamics of the quadrotor UAVs is derived based on the Newton-Euler laws in the body-fixed frame as ²⁶ :

$$\stackrel{·}{\mathrm{\Omega }}=R·\omega$$

(1)

$$\stackrel{·}{\omega }=I^{-1}·(-{\omega }^{\times }\times I\omega +M)+d$$

(2)

where the Euler angle vector, denoted as $\mathrm{\Omega }=[\varphi ,\theta ,\psi {]}^T$ , represents the roll angle, pitch angle, and yaw angle. The angular rate vector is $\omega =[p,q,r{]}^T$ , while $I=\mathrm{diag}(I_x,I_y,I_z)$ represents the symmetric positive-definite constant matrix representing the UAVs inertia. The control torque generated by the four rotors is denoted as $M={[M_x,M_y,M_z]}^T$ , and the lumped disturbance acting on the quadrotor UAVs, including model uncertainty and external bounded disturbances, is represented by $d={[d_1,d_2,d_3]}^T$ .

The rotational matrix $R$ is:

$$R=\left[\begin{array}{ccc}1\hfill & \sin \varphi \tan \theta \hfill & \cos \varphi \tan \theta \hfill \\ \multicolumn{1}{c}{0}& \cos \varphi \hfill & -\sin \varphi \hfill \\ \multicolumn{1}{c}{0}& \sin \varphi /\cos \varphi \hfill & \cos \varphi /\cos \theta \hfill \end{array}\right]$$

(3)

Furthermore, ${\omega }^{\times }$ represents the skew-symmetric matrix of $\omega$ :

$${\omega }^{\times }=\left[\begin{array}{ccc}0\hfill & -r\hfill & q\hfill \\ \multicolumn{1}{c}{r}& 0\hfill & -p\hfill \\ \multicolumn{1}{c}{-q}& p\hfill & 0\hfill \end{array}\right]$$

(4)

Consequently, the explicit form of quadrotor UAVs attitude dynamics is written as:

$$\begin{array}{c}\multicolumn{1}{c}{\{\begin{array}{c}\stackrel{·}{\mathrm{\Omega }}=\left[\begin{array}{ccc}1\hfill & \sin \varphi \tan \theta \hfill & \cos \varphi \tan \theta \hfill \\ \multicolumn{1}{c}{0}& \cos \varphi \hfill & -\sin \varphi \hfill \\ \multicolumn{1}{c}{0}& \sin \varphi /\cos \varphi \hfill & \cos \varphi /\cos \theta \hfill \end{array}\right]\omega \hfill \\ \multicolumn{1}{c}{\stackrel{·}{\omega }=\left[\begin{array}{c}\mathrm{qr}(I_y-I_z)/I_x\hfill \\ \multicolumn{1}{c}{\mathrm{pr}(I_z-I_x)/I_y}\\ \multicolumn{1}{c}{\mathrm{pq}(I_x-I_y)/I_z}\end{array}\right]+\left[\begin{array}{c}{M}_x/I_x\hfill \\ \multicolumn{1}{c}{M_y/I_y}\\ \multicolumn{1}{c}{M_z/I_z}\end{array}\right]+\left[\begin{array}{c}{d}_1\hfill \\ \multicolumn{1}{c}{d_2}\\ \multicolumn{1}{c}{d_3}\end{array}\right]}\end{array}}\end{array}$$

(5)

The primary aim of this article is to formulate and establish two adaptive neural network reinforcement learning (NN RL) controllers for UAVs. These controllers ensure the attainment of uniformly ultimately bounded signals throughout the closed-loop system, while also enabling the convergence of tracking errors to the proximity of zero without any potential collision hazards.

Radial basis function neural network

In the context of this article, the definition of the radial basis function neural network (RBFNN) function $\mathcal{F}(\varsigma )$ can be elucidated as follows:

$$\mathcal{F}(\varsigma )={\mathrm{\Theta }}^T\mathrm{\Phi }(\varsigma )$$

(6)

where $\varsigma$ is the system unknown of the estimated UAVs. $\mathrm{\Theta }=[{\mathrm{\Theta }}_1,\cdots ,{\mathrm{\Theta }}_l{]}^T$ represents the desired weight vector, and $\mathrm{\Phi }(\varsigma )=[{\mathrm{\Phi }}_1(\varsigma ),\cdots ,{\mathrm{\Phi }}_l(\varsigma ){]}^T$ denotes the basis function vector. The exact values of the weight vector remain unknown; however, they are bounded and adhere to the condition $\Vert \mathrm{\Theta }\Vert \le \overline{\mathrm{\Theta }}$ , where $\overline{\mathrm{\Theta }}$ represents an unknown constant greater than zero. Here, $l$ represents the number of nodes in the hidden layer. The Gaussian basis function is formulated as follows

$$\mathrm{\Phi }(Z)=\exp (-\frac{{(\varsigma -a)}^T(\varsigma -a)}{2b^2})$$

(7)

where $a$ is the coordinate vector representing the center points of the Gaussian basis functions, while $b$ represents the width of the Gaussian basis functions.

Numerous relevant publications have provided evidence of the capability of RBFNN to approximate diverse nonlinear functions across a confined domain ${\mathrm{\Omega }}_{\varsigma }$ :

$$\mathcal{F}(\varsigma )={\mathrm{\Theta }}^T\mathrm{\Phi }(\varsigma )+\epsilon (\varsigma )$$

(8)

where the ideal weight vector $\mathrm{\Theta }$ and the approximation error $\epsilon (\varsigma )$ are the variables of interest. It is assumed that the magnitude of the approximation error, denoted as $\Vert \epsilon (\varsigma )\Vert$ , is bounded by an unknown constant $\overline{\epsilon }>0$ .

Design of reinforcement learning control strategy

Within this section, an adaptive RL control scheme for discrete nonlinear systems of UAVs (5) is constructed as follows ²⁷ :

$$\begin{array}{c}\multicolumn{1}{c}{x_1(\hslash +2+1-1)=f_1\left({\overline{x}}_2(\hslash +2-1)\right)+g_1\left({\overline{x}}_2(\hslash +2-1)\right)}\\ \times x_2(\hslash +2-1)\hfill \\ \multicolumn{1}{c}{x_2(\hslash +1)=f_2({\overline{x}}_2(\hslash ))+g_2({\overline{x}}_2(\hslash ))u(\hslash )+d(\hslash )}\\ \multicolumn{1}{c}{y(\hslash )=x_1(\hslash )}\end{array}$$

(9)

where $x_1=\mathrm{\Omega }$ represents the system attitude angle of UAVs, and $x_2=\omega$ represents the system attitude angular rate of UAVs. In addition, $\hslash$ represents the discretized variables of the UAVs system using the forward differencing method. $u=M$ stands for the control moment vector of UAVs. Furthermore, $f_1=0,f_2=I^{-1}·(-{\omega }^{\times }\times I\omega ),g_1=R,g_2=I^{-1}$ . It is noted that the substitution of variables is intended to ensure the subsequent theoretical derivations are concise and clear.

Designs of critic NNs for avoiding obstacles

The purpose of the utility function denotes to a phenomenna that avoiding collision among UAVs, and keeping farther away from a obstacle, as shown in Figure 1. According to Refs,^22,25,27 by making inspection of the idea of few-shot learning, the utility cost function is proposed as follows:

$$M(\hslash )=\{\begin{array}{c}0\hfill \\ \multicolumn{1}{c}{1}\end{array}\begin{array}{c}\mathrm{if}\hfill \\ \multicolumn{1}{c}{\mathrm{if}}\end{array}\begin{array}{c}\left|{\rho }_d\right|\ge \epsilon \hfill \\ \multicolumn{1}{c}{\left|{\rho }_d\right|<\epsilon }\end{array}$$

(10)

where ${\rho }_d$ stands for the distance between UAVs ang obstacles. $\epsilon$ is the designed threshold of tracking effect, and $M(\hslash )$ is the index that reflects the level of effectiveness in tracking performance of the system.

Figure 1.

Reinforcement learning-based UAVs collision avoidance control framework.

Therefore, in order to encapsulate the overall performance of the policy over an extended period, the utility function for long-term policy evaluation is formulated in the following manner:

$$\begin{array}{c}\multicolumn{1}{c}{L_i(\hslash )={\pi }^N{M}_i(\hslash +1)+{\pi }^{N-1}{\pi }_i{M}_i(\hslash +2)+\cdots }\\ +{\pi }_i^{\hslash +1}{M}_i(N)\hfill \end{array}$$

(11)

where the symbol $\pi$ represents a positive design weighting parameter that fulfills the condition $0<\pi <1$ , while $N$ denotes the time horizon, indicating the final step in the simulation. Additionally, $L(\hslash )$ denotes the cumulative sum of all future performance indicators of the system. Equation (11) can also be equivalently expressed as $L(\hslash )=\underset{u(k)}{min}\{\pi L(\hslash -1)-{\pi }^{N+1}L\left(\hslash \right)\}$ , resembling the standard Bellman equation as described by Guo et al. ²²

Given the challenging nature of obtaining an exact value for $L(\hslash )$ , an alternative approach to obtain a suboptimal solution is by utilizing RBFNN to approximate it. Let us assume that:

$$L(\hslash )={\mathrm{\Theta }}_c^T{\mathrm{\Phi }}_c(\hslash )+{\epsilon }_c(\hslash )$$

(12)

where ${\mathrm{\Theta }}_c=[{\mathrm{\Theta }}_{c,1},\cdots ,{\mathrm{\Theta }}_{c,l}]T$ stands for the ideal weight vector, and the estimation error is remarked as ${\epsilon }_c(\hslash )$ , and suppose $\Vert {\mathrm{\Theta }}_c\left(\hslash \right)\Vert \le {\overline{\mathrm{\Theta }}}_c$ and $|{\epsilon }_c(\hslash )|\le {\overline{\epsilon }}_c$ , where ${\mathrm{\Theta }}_c>0$ and ${\overline{\epsilon }}_c>0$ are unknown constants. In addition, ${\mathrm{\Phi }}_c(\hslash )$ denotes the Gauss function.

Let us define the following Bellman error, which quantifies the discrepancy between the estimated value and the expected value in the Bellman equation, serving as a measure of the quality of the approximate solution:

$$E_c(\hslash )=\pi \hat{L}(\hslash )-[\hat{L}(\hslash -1)-M(\hslash )]$$

(13)

where $\hat{L}(\hslash )$ is the estimation of $L(\hslash )$ , and $J_c(\hslash )=(1/2)E_c^2(\hslash )$ is defined as a quadratic cost function to judge the control performacne of designed RL control algorithm. By means of the gradient descent method, the update law for ${\hat{\mathrm{\Theta }}}_c(\hslash )$ is designed as follows:

$${\hat{\mathrm{\Theta }}}_c(\hslash +1)={\hat{\mathrm{\Theta }}}_c(\hslash )-{\mu }_c{\mathrm{\Phi }}_c(\hslash )[\beta \hat{L}(\hslash )-\hat{L}(\hslash -1)+M(\hslash )]$$

(14)

In the upcoming content, a comprehensive exploration is undertaken to delve into the intricacies involved in formulating the backstepping RL mechanism for the purpose of designing advanced controllers:

• Step 1: Define $e_1(\hslash +2)=x_1(\hslash +2)-y_d(\hslash +2)$ and $e_2(\hslash +1)=x_2(\hslash +1)-{\alpha }_1(\hslash )$ from the UAVs system (9), and it yields

$$\begin{array}{l}{e}_1(\hslash +2)=g_1\left({\overline{x}}_2(\hslash +1)\right)[(\frac{f_1({\overline{x}}_2(\hslash +1)}{g_1({\overline{x}}_2(\hslash +1)}-\frac{y_d(\hslash +2)}{g_1({\overline{x}}_2(\hslash +1)}\\ +y_d(\hslash +2))+{\alpha }_1(\hslash )+e_2(\hslash +1)-y_d(\hslash +2)]\end{array}$$

(15)

where ${\alpha }_1(\hslash )$ is the virtual controller. ${\overline{x}}_2(\hslash )={[x_1(\hslash ),x_2(\hslash )]}^T$ .

Define

$${\mathrm{\Xi }}_1(\hslash )=-\frac{f_1({\overline{x}}_2(\hslash +1)}{g_1({\overline{x}}_2(\hslash +1)}+\frac{y_d(\hslash +2)}{g_1({\overline{x}}_2(\hslash +1)}-y_d(\hslash +2)$$

(16)

The elusive function ${\mathrm{\Xi }}_1(\hslash )$ is approximated through the utilization of a RBFNN, thereby enabling us to establish an estimation for this function:

$${\mathrm{\Xi }}_1(\hslash )={\mathrm{\Theta }}_1^T(\hslash ){\mathrm{\Phi }}_1({\varsigma }_1(\hslash ))+{\epsilon }_1(\hslash )$$

(17)

where ${\mathrm{\Theta }}_1(\hslash )=[{\mathrm{\Theta }}_{1,1}(\hslash ),{\mathrm{\Theta }}_{1,2}(\hslash ),\cdots ,{\mathrm{\Theta }}_{1,l}(\hslash ){]}^T$ is the ideal weight vector, and suppose $\Vert {\mathrm{\Theta }}_1(\hslash )\Vert \le {\overline{\mathrm{\Theta }}}_1$ and $|{\epsilon }_1(\hslash )|\le {\overline{\epsilon }}_1$ with ${\overline{\mathrm{\Theta }}}_1$ and ${\overline{\epsilon }}_1$ are unknown positive constants. ${\varsigma }_1(\hslash )={[x_1(\hslash +1),x_2(\hslash +1),y_d(\hslash +2)]}^T$ . The utilization of conventional adaptive methods unavoidably triggers computationally intensive real-time calculations, given the necessity to estimate multiple unknown weight parameters within the neural network. To address this concern, the adaptive updating mechanism for these multiple neural network weight parameters is constructed using the norm estimation strategy (MLP technologies).

By substituting (17) into (15), it yields

$$\begin{array}{l}{e}_1(\hslash +2)=g_1({\overline{x}}_2(\hslash +1))[-{\mathrm{\Theta }}_1{(\hslash )}_1^T{\mathrm{\Phi }}_1.({\varsigma }_1(\hslash ))-{\epsilon }_1(\hslash )\\ +{\alpha }_1(\hslash )+e_2(\hslash +1)-y_d(\hslash +2)]\end{array}$$

(18)

Let

$${\mathrm{\Upsilon }}_1(\hslash )={\mathrm{\Theta }}_1(\hslash {)}^T{\mathrm{\Phi }}_1({\aleph }_1(\hslash ))$$

(19)

where ${\aleph }_1(\hslash )={[x_1(\hslash +1),y_d(\hslash +2)]}^T$ . Through the incorporation of both the addition and subtraction of equation (19) to the right-hand side of equation (18), it becomes straightforward to derive the following expression:

$$\begin{array}{l}{e}_1(\hslash +2)=g_1\left({\overline{x}}_2(\hslash +1)\right)[-{\mathrm{\Theta }}_1^T(\hslash ){\mathrm{\Phi }}_1.({\varsigma }_1(\hslash ))-{\epsilon }_1(\hslash )\\ +{\mathrm{\Theta }}_1^T(\hslash ){\mathrm{\Phi }}_1\left({\aleph }_1(\hslash )\right)+{\alpha }_1(\hslash )-{\mathrm{\Theta }}_1^T(\hslash ){\mathrm{\Phi }}_1\left({\aleph }_1(\hslash )\right)\\ +e_2(\hslash +1)-y_d(\hslash +2)]\end{array}$$

(20)

The design of the virtual controller takes place in the following manner:

$${\alpha }_1(\hslash )={\hat{\mathrm{\Theta }}}_1^T(\hslash ){\mathrm{\Phi }}_1({\aleph }_1(\hslash ))+y_d(\hslash +2)$$

(21)

By substituting (21) into (20), one has

$$\begin{array}{l}{e}_1(\hslash +2)=g_1({\overline{x}}_2(\hslash +1))[-{\mathrm{\Theta }}_1^T(\hslash ){\mathrm{\Phi }}_1.({\varsigma }_1(\hslash ))-{\epsilon }_1(\hslash )\\ +{\mathrm{\Theta }}_1^T(\hslash ){\mathrm{\Phi }}_1({\aleph }_1(\hslash ))-{\tilde{\mathrm{\Theta }}}_1^T(\hslash ){\mathrm{\Phi }}_1({\aleph }_1(\hslash ))\\ +e_2(\hslash +1)]\end{array}$$

(22)

where ${\tilde{\mathrm{\Theta }}}_1(\hslash )={\hat{\mathrm{\Theta }}}_1(\hslash )-{\mathrm{\Theta }}_1(\hslash )$ .

At the time instance corresponding to $(\hslash +1)$ , the error variable (22) can be mathematically represented in the following manner:

$$\begin{array}{l}{e}_1(\hslash +1)=g_1({\overline{x}}_2(\hslash ))[-{\mathrm{\Theta }}_1^T({\hslash }_1){\mathrm{\Phi }}_1.({\varsigma }_1({\hslash }_1))-{\epsilon }_1({\hslash }_1)\\ +{\mathrm{\Theta }}_1^T({\hslash }_1){\mathrm{\Phi }}_1({\aleph }_1(\hslash ))\\ -{\tilde{\mathrm{\Theta }}}_1^T({\hslash }_1){\mathrm{\Phi }}_1({\aleph }_1({\hslash }_1))+e_2(\hslash )]\end{array}$$

(23)

where ${\hslash }_1=\hslash -1$ .

The definition of the strategic utility function encompasses a comprehensive evaluation framework that takes into account the strategic effectiveness and overall value derived from a particular course of collision-avoiding action, and we have

$$E_1(\hslash )={\hat{\mathrm{\Theta }}}_1^T({\hslash }_1){\mathrm{\Phi }}_1({\aleph }_1({\hslash }_1))+\hat{L}(\hslash )$$

(24)

Select the expense metric as $J_1(\hslash )=\frac{1}{2}{E}_1^2(\hslash )$ . Through the utilization of the gradient descent technique, the governing equation for updating ${\hat{W}}_1(\hslash )$ is deduced as follows:

$$\begin{array}{c}{\hat{\mathrm{\Theta }}}_1(\hslash +1)\hfill \\ ={\hat{\mathrm{\Theta }}}_1(\hslash )-{\mu }_1{\mathrm{\Phi }}_1({\aleph }_1({\hslash }_1))[{\hat{W}}_1^T({\hslash }_1){\mathrm{\Phi }}_1({\aleph }_1({\hslash }_1))+\hat{L}(\hslash )]\hfill \end{array}$$

(25)

• Step 2: Establish the error parameter $e_2(\hslash +1)=x_2(\hslash +1)-{\alpha }_1(\hslash )$ . The optimal regulator will be attained during the $2\mathrm{th}$ iteration. The expression for the error variable $e_2(\hslash +1)$ can be formulated as follows:

$$\begin{array}{c}{e}_2(\hslash +1)=g_2({\overline{x}}_2(k))\hfill \\ [\frac{f_2({\overline{x}}_2(\hslash ))}{g_2({\overline{x}}_2(\hslash ))}-\frac{{\alpha }_1(\hslash )}{g_2({\overline{x}}_2(\hslash ))+{\alpha }_1(\hslash )-{\alpha }_1(\hslash )+u(\hslash )}]+d(\hslash )\hfill \end{array}$$

(26)

Define

$${\mathrm{\Xi }}_2(\hslash )=-\frac{f_2({\overline{x}}_2(\hslash ))}{g_2({\overline{x}}_2(\hslash ))}+\frac{{\alpha }_1(\hslash )}{g_2({\overline{x}}_2(\hslash ))}-{\alpha }_1(\hslash )$$

(27)

Utilize RBFNN to approximate the unfamiliar function ${\mathrm{\Xi }}_2(\hslash )$ , and consider the following:

$${\mathrm{\Xi }}_2(\hslash )={\mathrm{\Theta }}_2^T(\hslash ){\mathrm{\Phi }}_2(Z_2(\hslash ))+{\epsilon }_2(\hslash )$$

(28)

where the optimal weight vector is denoted as ${\mathrm{\Theta }}_2(k)={[{\mathrm{\Theta }}_{2,1}(\hslash ),{\mathrm{\Theta }}_{2,2}(\hslash ),\cdots ,{\mathrm{\Theta }}_{2,l}(\hslash )]}^T$ . Let it be assumed that $\Vert {\mathrm{\Theta }}_2(\hslash )\Vert \le {\overline{\mathrm{\Theta }}}_2$ and $\Vert {\epsilon }_2(\hslash )\Vert \le {\overline{\epsilon }}_2$ , where ${\overline{\mathrm{\Theta }}}_2$ and ${\overline{\epsilon }}_2$ represent unspecified positive constants. ${\varsigma }_2(\hslash )={[x_1(\hslash ),x_2(\hslash ),y_d(\hslash +2)]}^T$ .

By replacing (28) in (26), it is possible to derive

$$\begin{array}{c}{e}_2(\hslash +1)=\hfill \\ g_2({\overline{x}}_2(\hslash ))[-{\mathrm{\Theta }}_2^T(\hslash ){\mathrm{\Phi }}_2({\aleph }_2(\hslash ))-{\epsilon }_2(\hslash )-{\alpha }_1(\hslash )+u(\hslash )]+d(\hslash )\hfill \end{array}$$

(29)

Design the controller as

$$u(\hslash )=-\sum _{j=1}^2{c}_j{e}_j(\hslash )+{\hat{\mathrm{\Theta }}}_2^T(\hslash ){\mathrm{\Phi }}_2({\aleph }_2(\hslash ))+{\alpha }_1(\hslash )$$

(30)

Through the process of substituting the equation (30) into the expression (29), we are able to arrive at the following result:

$$\begin{array}{c}\multicolumn{1}{c}{e_2(\hslash +1)=g_2({\overline{x}}_2(\hslash ))}\\ [{\tilde{\mathrm{\Theta }}}_2^T(\hslash ){\mathrm{\Phi }}_2({\aleph }_2(\hslash ))-{\epsilon }_2(\hslash )-\sum _{j=1}^2{c}_j{e}_j(\hslash )]+d(\hslash )\hfill \end{array}$$

(31)

where ${\tilde{\mathrm{\Theta }}}_2(\hslash )={\hat{\mathrm{\Theta }}}_2(\hslash )-{\mathrm{\Theta }}_2(\hslash )$ .

The strategic utility function is defined, providing a comprehensive framework for evaluating the overall effectiveness and value of strategic choices and decisions:

$$E_2(\hslash )={\hat{\mathrm{\Theta }}}_2^T(\hslash ){\mathrm{\Phi }}_2({\aleph }_2(\hslash ))+\hat{L}(\hslash )$$

(32)

Choose the cost function $J_2=(1/2)E_2^2(\hslash )$ . With the employment of the gradient descent method, a specific update law is formulated for ${\hat{\mathrm{\Theta }}}_2(k)$ , tailored to iteratively adjust and optimize its values:

$$\begin{array}{c}{\hat{\mathrm{\Theta }}}_2(k+1)=\hfill \\ {\hat{\mathrm{\Theta }}}_2(\hslash )-{\mu }_2{\mathrm{\Phi }}_2({\aleph }_2(\hslash ))[{\hat{\mathrm{\Theta }}}_2^T(\hslash ){\mathrm{\Phi }}_2({\aleph }_2(\hslash ))+\hat{L}(\hslash )]\hfill \end{array}$$

(33)

The previously mentioned design and analysis are consolidated and presented concisely in the subsequent theorem.

• Theorem 1: Under Assumption 1, the adaptive reinforcement learning (RL) control scheme is applied to the nonlinear UAVs system (9). This control scheme is consisted of control laws (21) and (30), updating laws of parameters (14), (25), and (33). Provided that the design parameters meet the conditions of $0<\beta <1$ , $0<{\mu }_c<\frac{1}{{\beta }^{2}l}$ , $0<{\mu }_i<\frac{1}{l}$ , and $c_i>0$ , there exist positive constants ${\kappa }_c$ , ${\kappa }_{W,i}$ , and ${\kappa }_{e,i}$ , ensuring that the adaptive RL control scheme ensures a guarantee of UUB for all signals encompassing variables and errors within the closed-loop system.

Proof: The Lyapunov function is determined by carefully selecting a suitable mathematical representation, resulting in

$$V(\hslash )=V_1(\hslash )+V_2(\hslash )+V_3(\hslash )+V_4(\hslash )+V_5(\hslash )$$

(34)

where

$$\begin{array}{c}{V}_1(\hslash )=\frac{{\kappa }_{e,1}}{4}{e}_i^2(\hslash ),V_2=\frac{{\kappa }_{e,2}}{5}{e}_2^2(\hslash )\hfill \\ V_3(\hslash )=\sum _{i=1}^2\frac{{\kappa }_{\mathrm{\Theta },i}}{{\mu }_i}\sum _{\lambda =1}^2{\tilde{\mathrm{\Theta }}}_i^T(\hslash +ƛ){\tilde{\mathrm{\Theta }}}_i(\hslash +ƛ)\hfill \\ V_4(\hslash )=\frac{{\kappa }_L}{{\mu }_L}{\tilde{\mathrm{\Theta }}}_L^T(\hslash ){\tilde{\mathrm{\Theta }}}_L(\hslash ),V_5(\hslash )=3{\kappa }_c[{\tilde{\mathrm{\Theta }}}_L^T(\hslash -1)\mathrm{\Phi }(\hslash -1)]\hfill \end{array}$$

where ${\tilde{\mathrm{\Theta }}}_L(\hslash )={\hat{\mathrm{\Theta }}}_L(\hslash )-{\mathrm{\Theta }}_L$ .

To proceed further, the derivative of $V_1(\hslash )$ with respect to the independent variable can be computed by employing the forward difference method, resulting in the following expression:

$$\begin{array}{c}\multicolumn{1}{c}{\mathrm{\Delta }{V}_1(\hslash )=\frac{{\kappa }_{e,1}}{4}{e}_1^2(\hslash +1)-\frac{{\kappa }_{e,1}}{4}{e}_1^2(\hslash )}\\ =\frac{{\kappa }_{e,1}}{4}{g}_1^2({\overline{x}}_2(\hslash ))[-{\mathrm{\Theta }}_1^T({\hslash }_1){\mathrm{\Phi }}_1(Z_1({\hslash }_1))-{\epsilon }_1({\hslash }_1)\hfill \\ +{\mathrm{\Theta }}_1^T({\hslash }_1){\mathrm{\Phi }}_1({\aleph }_1(\hslash )){-{\tilde{\mathrm{\Theta }}}_1^T({\hslash }_1){\mathrm{\Phi }}_1({\aleph }_1({\hslash }_1))+e_2(\hslash )]}^2\hfill \\ -\frac{{\kappa }_{e,1}}{4}{e}_1^2(\hslash )\hfill \end{array}$$

(35)

Invoke the Cauchychwarz inequality as

$${(x_1+x_2+\cdots +x_n)}^2\le n(x_1^2+\cdots +x_n^2)$$

(36)

Furthermore, by means of the Young’s inequality and $0<{\mathrm{\Phi }}_i^T(·){\mathrm{\Phi }}_i(·)<l$ of RBFNN combined with (23), it is obtained as

$$\begin{array}{c}-{\mathrm{\Theta }}_1^T({\hslash }_1){\mathrm{\Phi }}_1({\varsigma }_1({\hslash }_1))+{\mathrm{\Theta }}_1^T({\hslash }_1){\mathrm{\Phi }}_1({\aleph }_1(\hslash ))\hfill \\ \le |{\mathrm{\Theta }}_1^T({\hslash }_1){\mathrm{\Phi }}_1({\varsigma }_1({\hslash }_1))|+|{\mathrm{\Theta }}_1^T({\hslash }_1){\mathrm{\Phi }}_1({\aleph }_1(\hslash ))|\hfill \\ \le \frac{1}{2}{\mathrm{\Theta }}_1^T({\hslash }_1){\mathrm{\Theta }}_1({\hslash }_1)+\frac{1}{2}{\mathrm{\Theta }}_1^T({\hslash }_1){\mathrm{\Theta }}_1({\hslash }_1)\hfill \\ +\frac{1}{2}{\mathrm{\Phi }}_1^T({\varsigma }_1({\hslash }_1)){\mathrm{\Phi }}_1({\varsigma }_1({\hslash }_1))+\frac{1}{2}{\mathrm{\Phi }}_1^T({\aleph }_1({\hslash }_1)){\mathrm{\Phi }}_1({\aleph }_1({\hslash }_1))\hfill \\ \le {\overline{\mathrm{\Theta }}}_1^2+l\hfill \end{array}$$

(37)

To continue, $\mathrm{\Delta }{V}_1$ is further calculated as

$$\begin{array}{c}\mathrm{\Delta }{V}_1\le {\kappa }_{e,1}{g}_1^2({\overline{x}}_2(\hslash )){({\tilde{\mathrm{\Theta }}}_1^T({\hslash }_1){\mathrm{\Phi }}_1({\aleph }_1({\hslash }_1)))}^2\hfill \\ +{\kappa }_{e,1}{g}_1^2({\overline{x}}_2(\hslash ))e_2^2(\hslash )+{\kappa }_{e,1}{g}_1^2({\overline{x}}_2(\hslash )){\epsilon }_1^2({\hslash }_1)\hfill \\ +{\kappa }_{e,1}{g}_1^2({\overline{x}}_2(\hslash ))({\overline{\mathrm{\Theta }}}_1^2+l{)}^2-\frac{{\kappa }_{e,1}}{4}{e}_1^2(\hslash )\hfill \end{array}$$

(38)

By taking (31) and in mind, combined with (36), the $\mathrm{\Delta }{V}_2$ can be obtained as follows

$$\begin{array}{c}\mathrm{\Delta }{V}_2\le {\kappa }_{e,2}{g}_2^2({\overline{x}}_2(\hslash )){({\tilde{\mathrm{\Theta }}}_2^T(\hslash ){\mathrm{\Phi }}_2({\aleph }_2(\hslash )))}^2\hfill \\ +{\kappa }_{e,2}{g}_2^2({\overline{x}}_2(\hslash ))\sum _{i=1}^2{c}_i^2{e}_i^2(\hslash )+{\kappa }_{e,2}{g}_2^2({\overline{x}}_2(\hslash )){\epsilon }_2^2(\hslash )\hfill \\ -\frac{{\kappa }_{e,2}}{5}{e}_2^2(\hslash )+{\kappa }_{e,2}{\overline{d}}^2\hfill \end{array}$$

(39)

In what follows, from (25), (33), and (36), $\mathrm{\Delta }{V}_3(\hslash )$ can be further deduced as follows:

$$\begin{array}{c}\multicolumn{1}{c}{\mathrm{\Delta }{V}_3=V_3(\hslash +1)-V_3(\hslash )}\\ =\sum _{i=1}^2\frac{{\kappa }_{\mathrm{\Theta },i}}{{\mu }_i}({\tilde{\mathrm{\Theta }}}_i^T(\hslash +1){\tilde{\mathrm{\Theta }}}_i(\hslash +1)-{\tilde{\mathrm{\Theta }}}_i^T(\hslash ){\tilde{\mathrm{\Theta }}}_i(\hslash ))\hfill \\ \le \sum _{i=1}^2[{\kappa }_{\mathrm{\Theta },i}{\mu }_{i}l{[{\hat{\mathrm{\Theta }}}_i^T(\hslash ){\mathrm{\Phi }}_i({\aleph }_i(\hslash ))+\hat{L}(\hslash )]}^2\hfill \\ -2{\kappa }_{\mathrm{\Theta },i}{\tilde{\mathrm{\Theta }}}_i(\hslash )({\hat{\mathrm{\Theta }}}_i^T(\hslash ){\mathrm{\Phi }}_i({\aleph }_i(\hslash ))+\hat{L}(\hslash ))]\hfill \end{array}$$

(40)

By utilizing the Cauchychwarz inequality (36), it yields

$$\begin{array}{c}\multicolumn{1}{c}{\mathrm{\Delta }{V}_3\le -\sum _{i=1}^2{\kappa }_{\mathrm{\Theta },i}(1-{\mu }_{i}l)[{\hat{\mathrm{\Theta }}}_i^T(\hslash ){\mathrm{\Phi }}_i({\aleph }_i(\hslash ))+\hat{L}(\hslash ){]}^2}\\ -\sum _{i=1}^2{\kappa }_{\mathrm{\Theta },i}[{\tilde{\mathrm{\Theta }}}_i^T(\hslash ){\mathrm{\Phi }}_i({\aleph }_i(\hslash ))){]}^2+\sum _{i=1}^{2}2{\kappa }_{\mathrm{\Theta },i}l({\overline{\mathrm{\Theta }}}_i+{\overline{\mathrm{\Theta }}}_c{)}^2\hfill \\ +\sum _{i=1}^{2}2{\kappa }_{\mathrm{\Theta },i}({\tilde{\mathrm{\Theta }}}_c^T(\hslash ){\mathrm{\Phi }}_c({\aleph }_i(\hslash )){)}^2\hfill \end{array}$$

(41)

Similarly, by applying the forward difference technique, we can determine the derivative of $V_4(\hslash )$ with respect to the independent variable, leading to the subsequent equation:

$$\begin{array}{c}\multicolumn{1}{c}{\mathrm{\Delta }{V}_4=V_4(\hslash +1)-V_4(\hslash )}\\ \le -{\kappa }_c(1-{\mu }_c{\pi }^{2}l){[\pi \hat{L}(\hslash )-\hat{L}(\hslash -1)+M(\hslash )]}^2+3{\kappa }_c\hfill \\ -{\kappa }_c{\pi }^2{({\tilde{\mathrm{\Theta }}}_c^T(\hslash ){\mathrm{\Phi }}_c(\hslash ))}^2+3{\kappa }_c{({\tilde{\mathrm{\Theta }}}_c^T(\hslash -1){\mathrm{\Phi }}_c(\hslash -1))}^2\hfill \\ +3{\kappa }_{c}l{({\overline{\mathrm{\Theta }}}_c(1+\pi ))}^2\hfill \end{array}$$

(42)

By considering the explicit definition of forward difference, we can deduce the following:

$$\mathrm{\Delta }{V}_5(\hslash )=3{\kappa }_c{({\tilde{\mathrm{\Theta }}}_c^T(\hslash ){\mathrm{\Phi }}_c(\hslash ))}^2-3{\kappa }_c{({\tilde{\mathrm{\Theta }}}_c^T(\hslash -1){\mathrm{\Phi }}_c(\hslash -1))}^2$$

(43)

Ultimately, the discrepancy of the comprehensive Lyapunov function $V(\hslash )$ can be expressed as follows:

$$\begin{array}{l}\mathrm{\Delta }V\le -\sum _{i=1}^2{\kappa }_{\mathrm{\Theta },i}(1-{\mu }_{i}l){\left[{\widehat{\mathrm{\Theta }}}_i^T(\hslash ){\mathrm{\Phi }}_i({\aleph }_i(\hslash ))+\widehat{L}(\hslash )\right]}^2\\ -{\kappa }_c(1-{\mu }_c{\pi }^{2}l){\left[\pi \widehat{L}(\hslash )-\widehat{L}(\hslash -1)+M(\hslash )\right]}^2\\ +{\kappa }_{e,1}{g}_1^2\left({\overline{x}}_2(\hslash )\right)e_2^2(\hslash )-\frac{{\kappa }_{e,2}}{5}{e}_2^2(\hslash )\\ +{\kappa }_{e,2}{g}_2^2\left({\overline{x}}_2(\hslash )\right)c_2^2{e}_2^2(\hslash )-{\kappa }_c{\pi }^2{\left({\tilde{\mathrm{\Theta }}}_c^T(\hslash ){\mathrm{\Phi }}_c(\hslash )\right)}^2\\ +2\sum _{i=1}^2{\kappa }_{\mathrm{\Theta },i}{\left({\tilde{\mathrm{\Theta }}}_c^T(\hslash ){\mathrm{\Phi }}_c\left(\aleph (\hslash )\right)\right)}^2+3{\kappa }_c{\left({\tilde{\mathrm{\Theta }}}_c^T(\hslash ){\mathrm{\Phi }}_c(\hslash )\right)}^2\\ -\sum _{i=1}^2{\kappa }_{\mathrm{\Theta },i}{\left[{\tilde{\mathrm{\Theta }}}_i^T(\hslash ){\mathrm{\Phi }}_i({\aleph }_i(\hslash )))\right]}^2\\ +\sum _{i=1}^2{\kappa }_{e,i}{g}_i^2({\overline{x}}_2(\hslash )){\left({\tilde{\mathrm{\Theta }}}_i^T(\hslash ){\mathrm{\Phi }}_i(\aleph (\hslash ))\right)}^2\\ -\frac{{\kappa }_{e,1}}{4}{e}_1^2(\hslash )+{\kappa }_{e,2}{g}_2^2({\overline{x}}_2(\hslash ))c_1^2{e}_1^2(\hslash )\\ +\sum _{i=1}^2{\kappa }_{e,i}{g}_i^2({\overline{x}}_2(\hslash )){\epsilon }_i^2(\hslash )\\ +\sum _{i=1}^{2}2{\kappa }_{\mathrm{\Theta },i}l{\left({\overline{\mathrm{\Theta }}}_i+{\overline{\mathrm{\Theta }}}_c\right)}^2+{\kappa }_{e,1}{g}_1^2\left({\overline{x}}_2(\hslash )\right){\left({\overline{\mathrm{\Theta }}}_1^2+l\right)}^2\\ +{\kappa }_{e,2}{\overline{d}}^2+3{\kappa }_c+3{\kappa }_{c}l{({\overline{\mathrm{\Theta }}}_c(1+\pi ))}^2\end{array}$$

(44)

Choose the parameters $0<{\mu }_i<1/l$ and $0<{\mu }_c<1/{\pi }^{2}l$ , it infers that

$$\begin{array}{c}\multicolumn{1}{c}{\mathrm{\Delta }V\le -(\frac{{\kappa }_{e,1}}{4}-{\kappa }_{e,2}{\overline{g}}_2^2{c}_1^2)e_1^2(\hslash )}\\ -(\frac{{\kappa }_{e,2}}{5}-{\kappa }_{e,1}{\overline{g}}_1^2-{\kappa }_{e,2}{\overline{g}}_2^2{c}_2^2)e_2^2(\hslash )\hfill \\ -({\kappa }_c{\pi }^2-2\sum _{i=1}^2{\kappa }_{\mathrm{\Theta },i}-3{\kappa }_c){({\tilde{\mathrm{\Theta }}}_c^T(\hslash ){\mathrm{\Phi }}_c(\hslash ))}^2\hfill \\ -\sum _{i=1}^2({\kappa }_{\mathrm{\Theta },i}-{\kappa }_{e,i}{\overline{g}}_i^2){[{\tilde{\mathrm{\Theta }}}_i^T(\hslash ){\mathrm{\Phi }}_i({\aleph }_i(\hslash )))]}^2+\Pi \hfill \end{array}$$

(45)

By selecting the following parameters:

$$\begin{array}{c}\multicolumn{1}{c}{\frac{{\kappa }_{e,1}}{4}\ge {\kappa }_{e,2}{\overline{g}}_2^2{c}_1^2,\frac{{\kappa }_{e,2}}{5}\ge {\kappa }_{e,1}{\overline{g}}_1^2+{\kappa }_{e,2}{\overline{g}}_2^2{c}_2^2}\\ \multicolumn{1}{c}{{\kappa }_c{\pi }^2\ge 2\sum _{i=1}^2{\kappa }_{\mathrm{\Theta },i}-3{\kappa }_c,{\kappa }_{\mathrm{\Theta },i}\ge {\kappa }_{e,i}{\overline{g}}_i^2,i=1,2,}\end{array}$$

(46)

through meticulous scrutiny and thorough assessment, it can be inferred that the assertion of $\mathrm{\Delta }V(\hslash )<0$ is valid, indicating a reduction in the aggregate Lyapunov function. This deduction is contingent upon the fulfillment of the subsequent set of conditions^22,27:

$$\begin{array}{c}|e_1(\hslash )|\ge \sqrt{\Pi }/\sqrt{\frac{{\kappa }_{e,1}}{4}-{\kappa }_{e,2}{\overline{g}}_2^2{c}_1^2},\hfill \\ |e_2(\hslash )|\ge \sqrt{\Pi }/\sqrt{\frac{{\kappa }_{e,2}}{5}-{\kappa }_{e,1}{\overline{g}}_1^2-{\kappa }_{e,2}{\overline{g}}_2^2{c}_2^2}\hfill \\ |{\tilde{\mathrm{\Theta }}}_c^T(\hslash ){\mathrm{\Phi }}_c(\hslash )|\ge \sqrt{\Pi }/\sqrt{{\kappa }_c{\pi }^2-2\sum _{i=1}^2{\kappa }_{\mathrm{\Theta },i}-3{\kappa }_c},\hfill \\ |{\tilde{\mathrm{\Theta }}}_i^T(\hslash ){\mathrm{\Phi }}_i({\aleph }_i(\hslash )))|\ge \sqrt{\Pi }/\sqrt{{\kappa }_{\mathrm{\Theta },i}-{\kappa }_{e,i}{\overline{g}}_i^2},i=1,2\hfill \end{array}$$

(47)

Drawing upon the principles of the established Lyapunov extension theorem, it can be deduced that all signals within the closed-loop system are UUB, thereby substantiating the completion of the proof for Theorem 1.

Simulation

This section presents simulation and experimental results to illustrate the efficacy and resilience of the proposed approach. The simulation is conducted under the following conditions: The initial state vector of the quadrotor is denoted as $x_0=[0;0;0;0;0;0;0;0;0;0;0;0]$ . The quadrotor is subjected to certain constraints, including an upper limit on the propeller thrust at 15 and a lower limit at 0. Moreover, the values ${\delta }_i=0.85,i=1,2,3$ are employed. The sampling time is set to 0.2 s. In order to assess the effectiveness of the proposed control algorithm, simulations are conducted under the specified conditions. The performance evaluation of the proposed reinforcement learning (RL) control strategy primarily revolves around the attainment of a safe flight trajectory that enables the UAVs to navigate through moving static or dynamic obstacles while effectively mitigating the risk of collisions. Among them, in the static obstacle scenario, the starting point of UAVs is $[003{]}^T$ , and the endpoint is $[10,11,2{]}^T$ , as shown in Figure 2; while in the dynamic obstacle scenario, the starting point of UAVs is $[0,2,5{]}^T$ , and the endpoint is $[10,10,5.5{]}^T$ . This safe flight trajectory is visually represented in Figures 2 and 3.

Figure 2.

Position tracking trajectory of a quadrotor using the proposed RL algorithm for static obstacle.

Figure 3.

Position tracking trajectory of a quadrotor using the proposed RL algorithm for dynamic obstacle.

The simulation results provide compelling evidence of the effective collision-avoidance capabilities exhibited by the attainment of the target, thus serving the purpose of evaluating the health status of large-scale photovoltaic power generation while concurrently minimizing the potential risks of collisions (as depicted in Figures 2 and 3). Illustrated across Figures 2 and 3, the trajectory visualized by the red line demonstrates a remarkable ability to faithfully adhere to the intended path, and the blue line in Figure 3) represents the motion trajectory of dynamic obstacles, thereby ensuring the avoidance of collisions under both static and dynamic conditions. Furthermore, the simulation outcomes presented in Figure 3 substantiate the suitability of the proposed RL control methodology for a diverse array of tracking missions, while upholding compliance with stringent safety boundaries.

Conclusion

This study has made significant contributions to the control of UAVs. The investigation focused on transient performance, highlighting the importance of considering nonlinearities and ensuring stable maneuvering. By combining neural networks and reinforcement learning (RL), an innovative approach was developed for adaptive collision-free RL optimal control of UAVs with discrete-time systems. The introduction of the minimal learning parameter (MLP) reduced adaptive laws, improving computational efficiency without compromising performance. The proposed RL and MLP-based controllers ensured closed-loop system stability, demonstrated through UUB analysis. Overall, this research advances UAVs control strategies, emphasizing transient performance, neural networks, and RL techniques, with implications for safer and more efficient UAVs operations. In the future work, we will adopt the cooperative game indicator design concept to further enhance the control effect. In future work, building upon the autonomous obstacle avoidance control algorithm designed in this paper, further research will investigate autonomous cooperative obstacle avoidance control algorithms for UAVs under nonholonomic information constraints and scale variations.