Heuristic Q-learning based on experience replay for three-dimensional path planning of the unmanned aerial vehicle

Abstract

In order to solve the problem that the existing reinforcement learning algorithm is difficult to converge due to the excessive state space of the three-dimensional path planning of the unmanned aerial vehicle, this article proposes a reinforcement learning algorithm based on the heuristic function and the maximum average reward value of the experience replay mechanism. The knowledge of track performance is introduced to construct heuristic function to guide the unmanned aerial vehicles’ action selection and reduce the useless exploration. Experience replay mechanism based on maximum average reward increases the utilization rate of excellent samples and the convergence speed of the algorithm. The simulation results show that the proposed three-dimensional path planning algorithm has good learning efficiency, and the convergence speed and training performance are significantly improved.

Introduction

Unmanned aerial vehicle (UAV) has attracted wide attention from scholars all over the world in past decade. UAV has the advantages of small size, low cost, convenient use, low requirements for the operational environment, flexible, no casualty risk, and so on. It is widely used in aerial photography, plant protection, express transportation, disaster rescue, surveying and mapping, power inspection, reconnaissance, and other fields. Path planning is one of the key technologies for UAVs to accomplish the above tasks. Path planning refers to finding an optimal or feasible trajectory from the starting point to the target point under given space and constraints. UAV path planning is an NP complex problem with multiple constraints, which include fuel consumption and maneuvering constraints, terrain obstacles, threat information, and so on. Many scholars have done a lot of work in path planning. ${\mathrm{A}}^{*}$ algorithms ¹ are widely used in path planning due to its ease of implementation and high efficiency, but it is difficult to ensure that an optimal path is found. Artificial potential field method ² realizes path planning by establishing the gravitational field and repulsion field function, but it is easy to fall into the local optimal solution. The Voronoi diagram ³ has a good performance in obstacle avoidance, but it needs to describe the obstacle information with geometric structure, which is difficult to adapt to large dynamic environment. Swarm intelligence algorithm ⁴ (such as ant colony algorithm, particle swarm algorithm) has the advantages of simple structure, high precision, fast convergence, and so on. However, there are still complex calculation problems in high-dimensional and complex planning space.

Reinforcement learning is an important method in the field of machine learning. ⁵ Unlike supervised learning and unsupervised learning, reinforcement learning relies on the agent to continuously interact with the environment, and each interaction receives an evaluative feedback signal in return for learning the optimal behavior by maximizing its cumulative reward. Theoretically, reinforcement learning does not depend on the exact model. Whatever exploration and utilization strategy is adopted, it will converge to the optimal value after a long enough time. But for a practical problem, when the state space is large and each state can perform more actions, there will be a very large state-action space. In order to accelerate the convergence speed of reinforcement learning algorithm, researchers have proposed several effective methods. Hengst ⁶ uses hierarchical reinforcement learning to decompose large-scale reinforcement learning problems into several sub-problems, which reduces the state space of the problem. However, reasonable hierarchical processing is a challenging task and is difficult to achieve. Andrew et al. ⁷ introduced shaping function into reinforcement learning and added heuristic value to the returns of agents, which effectively improved the convergence speed. Asmuth et al. ⁸ used potential field function as a priori knowledge to enlighten the reinforcement learning process and proved the effectiveness of the algorithm.

Based on the existing reinforcement learning algorithm for path planning, this article proposes a three-dimensional path planning method for UAV based on heuristic return function and maximum average reward experience replay mechanism. First, the state space of UAV is discretized to reduce the scale of path planning problem. Then, a heuristic reward function is constructed based on the maneuverability of UAV, fuel consumption, terrain obstacles, flight altitude, and other factors to improve the convergence speed of the algorithm. The experience replay mechanism is improved, and the importance of samples is evaluated by the maximum average return value at a small computational cost. Finally, the validity of the method is verified by the three-dimensional path planning simulation experiment of UAV, which achieves the effective approximation of the value function, and has good learning efficiency and generalization performance. The convergence speed and training performance are obviously improved.

Problem formulation

The state space of UAV in real environment is continuous, which greatly increases the difficulty of problem-solving. Therefore, three-dimensional discretization of planning space should be carried out first, so the search space of path planning problem is reduced to a discrete set of spatial nodes; each node $v_k$ represents the three-dimensional coordinates $(x_k,y_k,z_k)$ of UAV. The UAV path planning problem can be described as finding several nodes in the space node set, which minimizes the total cost function of UAV flying along the path composed of these nodes. Let the set of all discrete state space nodes set V, as follows

$$V=\{v_1,v_2,v_3,\dots ,v_n|v_k=(x_k,y_k,z_k)\}$$

The set of all flight paths, including the starting point and the target point, is represented by L

$$L=\{L_1,L_2,L_3,\dots ,L_m\}$$

The cost from state space node $v_i$ to state space node $v_j$ can be expressed in $c(v_i,v_j)$ , and the UAV path planning problem can be described in mathematical language as follows

$$\begin{array}{c}C(L_k)=\sum _{(v_i,v_j\in L_k)}c(v_i,v_j)\hfill \\ s.t.L_k\in L,v_i,v_j\in V\hfill \end{array}$$

(1)

where $C(L_k)$ represents the total cost of the available path $L_k$ .

Q-learning

Algorithm principle

The research of reinforcement learning is based on the theoretical framework of Markov decision process (MDP). The state and reward of the next moment only depend on the current state. The MDP can be represented by a quaternion $(S,A,p_{\mathrm{sa}},R)$ :

1. S is the set of states, and any state $s\in S$ , $s_t$ represents the state of step t of UAV, the state $s_t$ only includes three-dimensional position coordinates $(x_t,y_t,z_t)$ .

2. A is the set of actions, and any action $a\in A$ , and $a_t$ represents the actions that UAV executes in step t. The action of UAV can be composed of horizontal and vertical actions. The horizontal actions include left turning 45°, direct flying, and right turning 45°. The vertical actions include climbing 45°, horizontal flying, and descending 45°. Therefore, through different combinations of horizontal and vertical actions, UAV’s action space includes nine kinds of actions.

3. $p_{\mathrm{sa}}$ represents the probability distribution of state transition of UAV after executing an action $a_t$ in state $s_t$ , and the probability of transition to state $s_{t+1}$ is $p_{\mathrm{sa}}(s_{t+1}|s_t,a_t)$ . For UAV three-dimensional path planning, the probability of UAV moving to other states after executing its action $a_t$ in the state $s_t$ is determined $p_{\mathrm{sa}}(s_{t+1}|s_t,a_t)=1$ .

4. R is the reward function, which represents the immediate reward that UAV receives when it executes an action $a_t$ in a state $s_t$ and transfers to a state $s_{t+1}$ . It can also be expressed as $R(s_t,a_t,s_{t+1})=r_t$ .

The model of reinforcement learning can be described as UAV–environment interaction in Figure 1. The UAV chooses an action $a_t$ in the state $s_t$ . It then transfers to the next state $s_{t+1}$ according to the probability $p_{\mathrm{sa}}$ of state transition and gets a reward value. Then, the UAV executes another action $a_{t+1}$ according to state $s_{t+1}$ and repeats the above process.

Figure 1.

Reinforcement learning model.

The goal of reinforcement learning is to learn a good strategy to maximize future cumulative rewards in sequential decision-making. The sum of discount cumulative rewards is called expected reward $G(t)$

$$G\left(t\right)=r_{t+1}+\gamma r_{t+2}+{\gamma }^2{r}_{t+3}+{\gamma }^3{r}_{t+4}+\cdots =\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k+1}$$

(2)

where $\gamma$ represents the discount factor, the range of values is between 0 and 1. The closer to 0, the more important the present reward is, and the closer to 1, the more important the future reward is.

The state-action value function $Q^{\pi }(s,a)$ represents the mathematic expectation of reward that UAV can obtain after choosing the action a in the state s and executing the strategy $\pi$ all the time. It can be written as follows

$$Q^{\pi }(s,a)=E_{\pi }[\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k+1}|s_t=s,a_t=a]$$

(3)

Q-learning algorithm is a model-free offline strategy reinforcement learning control algorithm, ⁹ which has good convergence in discrete MDP. The state-action value function $Q(s_t,a_t)$ is updated according to Bellman equation ¹⁰

$$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha (r_t+\underset{a\prime \in A}{max}Q(s_{t+1},a\prime )-Q_t(s_t,a_t))$$

(4)

where $\alpha$ is the learning rate, $0<\alpha <1$ .

Training strategy

Because of the huge state space of UAV’s three-dimensional path planning, it is difficult to record all state-action value functions in tabular form. At present, the common method is to approximate the optimal action-value function with neural network and adjust the weight parameters of neural network by reducing the mean square error of Bellman equation. However, the convergence of this method is poor. Experiential playback and target separation can accelerate the convergence speed of the algorithm. ¹¹

Experience replay mechanism is proposed by Lin. ¹² A fixed-length cached sample data set $D_t=\{d_1,d_2,d_3,\dots ,d_t\},d_t=\{s_t,a_t,r_t,s_{t+1}\}$ is set up. With the increase in data samples, the old data in the data set are updated with new data. Episode $\mathrm{ep}{s}_k$ refers to the process of UAV from the starting point to the collision or arrival at the target point. Every n episode, m samples are randomly selected from the cached sample data set and put into Q-learning algorithm. Experience replay mechanism has two main advantages:

1. Each sample $d_t$ may be used many times, which improves the efficiency of data utilization.

2. It breaks the correlation of continuous samples in traditional Q-learning algorithm and helps to prevent the algorithm from falling into local optimum.

Minh et al. ¹³ proposed to use two neural networks for learning, one is target neural network predicting $Q(s\prime ,a;{\theta }_j^{\mathrm{target}})$ , and the other is estimation neural network predicting $Q(s,a;{\theta }_i^{\mathrm{est}})$ . Bellman equation can be expressed as follows

$$\mathrm{la}{b}_i=Q(s,a;{\theta }_i^{\mathrm{est}})+\alpha (r_t+\underset{a\prime \in A}{max}Q(s\prime ,a;{\theta }_j^{\mathrm{target}})-Q(s,a;{\theta }_i^{\mathrm{est}}))$$

(5)

When the estimated neural network is trained, the loss function of the neural network can be written as follows

$$J\left({\theta }_i^{\mathrm{eval}}\right)=\frac{1}{2}{(\mathrm{la}{b}_i-Q(s,a;{\theta }_i^{\mathrm{est}}))}^2$$

(6)

The estimated neural network is continuously updated. When it is updated to a certain number of times o, the parameters of the estimated neural network are copied to the target neural network. Target neural network delays the influence of parameter updating, eliminates the correlation between the estimated value and the target value, and reduces the possibility of non-convergence of the algorithm.

Action selection strategy

In order to ensure good convergence, action selection strategies need to consider both exploration and application. The commonly used action selection strategy is $\epsilon -\mathrm{greedy}$ , which uses the probability of $1-\epsilon$ to select the maximum action in the state and the probability of $\epsilon$ to randomly select any action in the available action space

$$\pi \left(s\right)=\{\begin{array}{cc}\mathrm{random}\mathrm{choice}a\hfill & \epsilon \hfill \\ \multicolumn{1}{c}{\underset{a\in A}{\arg max}\left(Q(s,a)\right)}& 1-\epsilon \hfill \end{array}$$

(7)

Algorithm 1: Q-learning
Input: discount factor $\gamma$ , learning rate $\alpha$ , exploring factor $\epsilon$
Neural network training parameters n, o, m, u, number of training times N, starting point $s_0$ , target point $s_{\mathrm{target}}$
Initialize: $Q(s,a;{\theta }_0^{\mathrm{est}}),Q(s,a;{\theta }_0^{t\arg \mathrm{et}}),(\forall s\in S,a\in A)$
For k = 1:N do
Initial state $s_0$
For t = 1, 2, 3, … do
Choose $a_t$ from A using equation (7)
Take action $a_t$ that leads to new state $s_{t+1}$
Calculate return value $r_t$ by
$r_t=\{\begin{array}{cc}1\hfill & \mathrm{beyond}\mathrm{or}\mathrm{collision}\hfill \\ \multicolumn{1}{c}{-1}& \mathrm{arrive}\mathrm{the}\mathrm{goal}\hfill \\ \multicolumn{1}{c}{0}& \mathrm{otherwise}\hfill \end{array}$
$\mathrm{ep}{s}_k\leftarrow [s_t,a_t,r_t,s_{t+1}]$
$s_t\leftarrow s_{t+1}$
Until $r_t\ne 0$
If rem(k, n) = 0
Choose m samples to update $Q(s,a;{\theta }_i^{\mathrm{est}})$ using equation (6)
If rem(k, o) = 0
${\theta }_i^{\mathrm{target}}\leftarrow {\theta }_i^{\mathrm{est}}$

Heuristic Q-learning

In order to improve the convergence speed of Q-learning, this article proposes a heuristic Q-learning method by improving the reward function and action selection strategy.

Reward function

Traditional Q-learning mostly uses sparse reward function. For path planning problem, if UAV reaches the target point, it will give a positive reward value, and if it collides with obstacles or exceeds the state space range, it will generate a negative penalty value. In other cases, the return value is generally 0. Sparse reward function has the advantages of simple calculation and easy design. But for UAV three-dimensional path planning problem, the state space is very large, the reward value of most actions is 0, and the probability of finding meaningful reward value is relatively small which leads to the slow convergence speed of learning algorithm. In order to overcome the shortcomings of sparse return function, a heuristic reward function $R(s_t,a_t,s_{t+1})$ is constructed by introducing more factors affecting path performance, such as UAV maneuverability, fuel consumption, terrain threat, distance to target, and flight altitude

$$r_t=R(s_t,a_t,s_{t+1})={\omega }_1{p}_1+{\omega }_2{p}_2+{\omega }_3{p}_3+{\omega }_4{p}_4+{\omega }_5{p}_5$$

(8)

where ${\omega }_1~{\omega }_5$ is the influence weight of each factor, and $p_1~p_5$ is the evaluation factor of path performance. The concrete connotation and calculation method are as follows:

1. p ₁ represents the fuel consumption cost of the UAV from state $s_t$ to state $s_{t+1}$ . The fuel cost is to ensure that the UAV reaches its destination in as short a time as possible

   $$p_1=-\sqrt{{(x_t-x_{t+1})}^2+{(y_t-y_{t+1})}^2+{(z_t-z_{t+1})}^2}$$

   (9)

2. In the artificial potential field method, the target point will attract the UAV. According to this idea, this article constructs $p_2$ . The closer to the target point, the more returns will be obtained, which will provide direction guidance for the choice of UAV operation

   $$\mathrm{em}=\sqrt{x_{\mathrm{target}}^2+y_{\mathrm{target}}^2+z_{\mathrm{target}}^2}-\sqrt{{(x_t-x_{\mathrm{target}})}^2+{(y_t-y_{\mathrm{target}})}^2+{(z_t-z_{\mathrm{target}})}^2}$$

   (10)
   $$p2={\log }_{10}(\mathrm{em}+\delta )$$

   (11)

   where $(x_{\mathrm{target}},y_{\mathrm{target}},z_{\mathrm{target}})$ represents the three-dimensional coordinates of the target point, $\delta$ is a constant greater than 0, which prevents calculation errors when em is equal to 0.

3. p ₃ is the cost of UAV deviating from cruise altitude $z_0$ , which is designed to ensure UAV flying at cruise altitude as far as possible

   $$p_3=-|z_t-z_0|$$

   (12)

4. p ₄ is the cost of collision between UAV and obstacle or beyond the scope of state space

   $$p_4=\{\begin{array}{cc}{R}_0\hfill & \mathrm{beyond}\mathrm{or}\mathrm{collision}\hfill \\ \multicolumn{1}{c}{0}& \mathrm{otherwise}\hfill \end{array}$$

   (13)

   where $R_0$ is a large negative constant.

5. p ₅ is the reward value of the UAV when it reaches the target point. A larger reward value can make the algorithm converge quickly

   $$p_5=\{\begin{array}{cc}{R}_1\hfill & \mathrm{arrive}\mathrm{the}\mathrm{goal}\hfill \\ \multicolumn{1}{c}{0}& \mathrm{otherwise}\hfill \end{array}$$

   (14)

   where $R_1$ is a large positive constant.

Action selection strategy

The key point of Q-learning algorithm convergence is to explore and utilize the balance problem and choose the appropriate action selection strategy which can not only make UAV explore the environment fully to avoid falling into local optimal solution but also make the learning algorithm converge faster. The optimal strategy of Q-learning is usually based on the estimation of state-action value function $Q(s,a)$ . However, in the initial stage of algorithm learning, UAV cannot get enough information about the environment, and the optimal estimation of state-action value function is not accurate, which often makes UAV collide with obstacles, seriously affecting the learning efficiency of the algorithm.

Therefore, this article designs an evaluation function $E(s,a)$ to assist the state-action value function $Q(s,a)$ to select the action. The meaning of $E(s,a)$ is that the UAV performs nine actions in the action space separately in the state s. When the UAV collides with the obstacle after the action, it returns to −100, and if there is no collision, it returns to 0

$$\underset{a\in A}{E(s,a)}=\{\begin{array}{cc}-100\hfill & \mathrm{collision}\hfill \\ \multicolumn{1}{c}{0}& \mathrm{otherwise}\hfill \end{array}$$

(15)

If $E(s,a_i)$ is −100, then $a_i$ is removed from the action space in the state s. The evaluation function $E(s,a)$ can reduce the available action space of UAV in the state s and avoid a large number of collisions in the initial stage. The action selection strategy of heuristic Q-learning is $\epsilon \mathrm{-}\mathrm{greedy}$ , which chooses the action a when $(Q(s,a)-E(s,a))$ takes the maximum in the state s with probability $1-\epsilon$ and randomly chooses any action in the available action space with probability $\epsilon$

$$\pi \left(s\right)=\{\begin{array}{cc}\mathrm{random}\mathrm{choice}a\hfill & \epsilon \hfill \\ \multicolumn{1}{c}{\underset{a\in A}{\arg max}(Q(s,a)-E(s,a))}& 1-\epsilon \hfill \end{array}$$

(16)

In the early stage of learning, UAV is ignorant of environmental information, so action selection strategies should pay more attention to exploration. With the deepening of learning, the reliability of state-action value function is getting higher, and action selection strategies should be paid more attention to and utilized. So $\epsilon$ is not a fixed value, but a relatively big value at the beginning, and then gradually reduces to nearly 0 with the increase of learning times

$$\epsilon =\epsilon -{\epsilon }_{\mathrm{increment}}({\epsilon }_{min}<\epsilon <{\epsilon }_0)$$

(17)

where ${\epsilon }_0$ is the initial value of exploratory factor. ${\epsilon }_{min}$ is the minimum value of exploratory factor. ${\epsilon }_{\mathrm{increment}}$ is the increment of exploratory factor.

Algorithm 2: Heuristic Q-learning
Input: discount factor $\gamma$ , learning rate $\alpha$ , exploring factor ${\epsilon }_0$ , ${\epsilon }_{min}$ , ${\epsilon }_{\mathrm{increment}}$
Neural network training parameters m, n, o, u, number of training times N, reward parameters ${\omega }_1~{\omega }_5$ , $R_0$ , $R_1$ , $\delta$ , $z_0$ , starting point $s_0$ , target point $s_{\mathrm{target}}$
Initialize: $Q(s,a;{\theta }_0^{\mathrm{est}}),Q(s,a;{\theta }_0^{\mathrm{target}}),(\forall s\in S,a\in A)$
For k = 1:N do
Initial state $s_0$
For t = 1, 2, 3, … do
Choose $a_t$ using equation (16)
Take action $a_t$ that leads to new state $s_{t+1}$
Calculate return value $r_t$ using equation (8)
$\mathrm{ep}{s}_k\leftarrow [s_t,a_t,r_t,s_{t+1}]$
$s_t\leftarrow s_{t+1}$
until $p_4+p_5\ne 0$
If rem(k, n) = 0
Choose m samples to update $Q_k(s,a;{\theta }_0^{\mathrm{est}})$ using equation (6)
If rem(k, o) = 0
${\theta }_i^{\mathrm{target}}\leftarrow {\theta }_i^{\mathrm{est}}$

MARER Q-learning

The experience replay mechanism has the drawback that the sampling is randomly performed in the buffered sample data set, and the quality of the sample is neglected. The sample that the UAV has reached the target point after thousands of explorations may not have the Q-learning algorithm selected. In order to improve the learning efficiency, Schaul et al. ¹⁴ proposed the sampling strategy of prioritized replay and sorted the samples according to their importance. However, the algorithm needs to continuously sort the samples in the data set, which greatly increases the computational complexity.

In order to further improve the efficiency of the algorithm and reduce the complexity of the algorithm, this article proposes a preferred cached experience replay mechanism using the maximum average reward value (MARER Q-learning) based on heuristic Q-learning:

• Calculate the average of reward values $r_k^{\mathrm{ave}}$ for the episode $\mathrm{ep}{s}_k$ , as follows

  $$r_k^{\mathrm{ave}}=E\left(\sum _{r_i\in \mathrm{ep}{s}_k}{r}_i\right)$$

  (18)

• Then update the maximum average reward value $r_{max}^{\mathrm{ave}}$

  $$r_{max}^{\mathrm{ave}}=\{\begin{array}{cc}{r}_k^{\mathrm{ave}}\hfill & r_k^{\mathrm{ave}}>r_{max}^{\mathrm{ave}}\hfill \\ \multicolumn{1}{c}{r_{max}^{\mathrm{ave}}}& r_k^{\mathrm{ave}}\le r_{max}^{\mathrm{ave}}\hfill \end{array}$$

  (19)

If the maximum average reward value is updated, indicating that there is a good sample in this episode, all the samples included in this episode are directly put into the algorithm for learning, and the samples in this episode are copied three times and then put into the cached sample data set, which improves the proportion of excellent samples. Then, algorithm randomly selects m samples from the cached sample data set every n episode into the Q-learning algorithm for learning.

Algorithm 3: MARER Q-learning
Input: discount factor $\gamma$ , learning rate $\alpha$ , exploring factor ${\epsilon }_0$ , ${\epsilon }_{min}$ , ${\epsilon }_{\mathrm{increment}}$
Neural network training parameters m, n, o, u, number of training times N, reward parameters ${\omega }_1~{\omega }_5$ , $R_0$ , $R_1$ , $\delta$ , $z_0$ , starting point $s_0$ , target point $s_{\mathrm{target}}$
Initialize: $Q(s,a;{\theta }_0^{\mathrm{est}}),Q(s,a;{\theta }_0^{\mathrm{target}}),(\forall s\in S,a\in A)$ , $r_{max}^{\mathrm{ave}}$
For k = 1:N do
Initial state $s_0$ , $\mathrm{ep}{s}_k=\left[\right]$
For t = 1, 2, 3, … do
Choose $a_t$ from A using equation (16)
Take action $a_t$ that leads to new state $s_{t+1}$
Calculate return value $r_t$ using equation (8)
$\mathrm{ep}{s}_k\leftarrow [s_t,a_t,r_t,s_{t+1}]$
$s_t\leftarrow s_{t+1}$
Until $p_4+p_5\ne 0$
Calculate the average reward $r_k^{\mathrm{ave}}$ using equation (18)
If $r_k^{\mathrm{ave}}>r_{max}^{\mathrm{ave}}$
Update $Q(s,a;{\theta }_0^{\mathrm{est}})$ using equation (19) with sample set $\mathrm{ep}{s}_k$
Copy $\mathrm{ep}{s}_k$ three times and put it in $D_t:D_t\leftarrow \mathrm{ep}{s}_k\times 3$
Update maximum average reward $r_{max}^{\mathrm{ave}}$ using equation (19)
If rem(k, n) = 0
Choose m samples to update $Q_k(s,a;{\theta }_0^{\mathrm{est}})$ using equation (6)

Simulation results

Assume that the mission area of the drone is 200 km × 100 km × 250 m, and a representative topographic map is generated according to typical terrain features. The discrete step in the horizontal direction is 5 km, and the discrete step in the height direction is 5 m. The discrete space of the three-dimensional path planning of the machine is 40 × 20 × 50, and there are 40 × 20 × 50 × 9 state-action values.

The expression for the terrain model used for the simulation is as follows

$$Z(X,Y)=\sum _{i=1}^8{h}_i{e}^{-{\left(\frac{X-a_i}{c_i}\right)}^2-{\left(\frac{Y-b_i}{c_i}\right)}^2}$$

(20)

$(X,Y)$ is the coordinate of the horizontal direction, $Z(X,Y)$ is the height of the mountain in this horizontal coordinate. $h_i$ determines the maximum height of the mountain, $c_i$ determines the horizontal range of the mountain, and $(a_i,b_i)$ determines the horizontal coordinates of the mountain (Figure 2)

$$\begin{array}{c}h=[125,150,250,200,150,150,150,150]\hfill \\ c=[200,300,200,100,300,300,200,100]\hfill \\ a_i=[25,40,50,80,75,140,140,175]\hfill \\ b_i=[25,80,45,70,11,60,15,30]\hfill \end{array}$$

Figure 2.

Three-dimensional topographic map.

Simulation experiment parameters are as follows:

1. Reward parameters ${\omega }_1=0.05,{\omega }_2=0.2,{\omega }_3=0.003,{\omega }_4=0.01,{\omega }_5=0.01$ , $R_0=-100$ , $R_1=100$ , Cruising altitude, $z_0=30$ , bias $\delta =0.01$ .

2. Discount factor $\gamma =0.9$ , learning rate $\alpha =0.01$ , exploring factor ${\epsilon }_0=0.7$ , ${\epsilon }_{max}=0.95$ , $b_i=[25,80,45,70,11,60,15,30]$ , ${\epsilon }_{\mathrm{increment}}=0.00001$ .

3. Neural network training interval $n=5$ , neural network replacement interval $o=100$ , number of training samples per batch $m=200$ , size of sample data set $u=1000$ , number of training times N = 30,000.

4. Position coordinates of starting point $s_0=(10\mathrm{km},1\mathrm{km},30\mathrm{m})$ , position coordinates of target point $s_{\mathrm{target}}=(200\mathrm{km},30\mathrm{km},30\mathrm{m})$ .

5. The maximum number of steps per learning episode is 200.

By statistically reaching the episode of the target point successfully, as shown in Figures 3 and 4, we can clearly see the difference in convergence speed of the three reinforcement learning methods. Heuristic Q-learning begins to converge after about 1700 episodes. MARER Q-learning converges after about 1300 episodes. However, Q-learning has not reached the target point after 2000 episodes. The fastest convergence among the three algorithms is the MARER Q-learning algorithm, while the convergence of Q-learning is the slowest. Heuristic Q-learning constructs a reward function by introducing path information such as height and distance, so that the UAV can have a deeper understanding of the environment and actively eliminates adverse actions to reduce the motion search space, thereby accelerating the convergence speed of the algorithm. Based on the heuristic Q-learning, MARER Q-learning evaluates the pros and cons of the sample by the maximum average return value and makes the algorithm select the excellent samples in a targeted manner, thus achieving more excellent results.

Figure 3.

The number of episode of successful reach to the target point.

Figure 4.

Average reward per episode: (a) heuristic Q-learning and (b) MARER Q-learning.

The optimal three-dimensional path given by heuristic Q-learning is shown in Figure 5, and the projection of the optimal trajectory (heuristic Q-learning) on the height profile and the horizontal profile is shown in Figure 6. The optimal path given by MARER Q-learning is shown in Figure 7, and the projection of the optimal path (MARER Q-learning) on the height profile and the horizontal profile is shown in Figure 8. By comparing the vertical plane of Figures 6 and 8, it can be found that the flight height of the latter is lower, and the latter has better obstacle avoidance ability; by comparing the horizontal plane of the two, it can be found that the track of the latter is more straight and has better economic performance. In summary, the MARER Q-learning algorithm is better than the other algorithms in terms of convergence speed and planning results.

Figure 5.

Three-dimensional path map of UAV (heuristic Q-learning).

Figure 6.

Three-dimensional path profile of UAV (heuristic Q-learning): (a) vertical plane and (b) horizontal plane.

Figure 7.

Three-dimensional path map of UAV (MARER Q-learning).

Figure 8.

Three-dimensional path profile of UAV (MARER Q-learning): (a) vertical plane and (b) horizontal plane.

It can be seen from the figure that the two reinforcement learning methods can successfully find the lower position of the two mountain joints and pass through them, climb up after encountering obstacles, and the altitude will drop to cruise altitude after overturning obstacles. The path planning of MARER Q-learning is better than that of heuristic Q-learning. The three-dimensional path planning of MARER Q-learning is mostly flat, basically the best path from the starting point to the target point.

Conclusion

Traditional Q-learning methods are less efficient. When the scale of state space increases linearly, the complexity of the problem increases exponentially, so the traditional reinforcement learning method is difficult to solve the problem of UAV three-dimensional path planning. This article proposes a Q-learning algorithm based on the heuristic function and the maximum average reward value of the experience replay mechanism. By synthetically considering the constraints of UAV path planning, a heuristic function is constructed to guide the learning behavior of UAV effectively, which can break away from blind exploration to a certain extent and improve the learning efficiency. The improved empirical playback mechanism greatly improves the convergence speed of the algorithm with a small computational cost. The simulation results show that the three-dimensional trajectory obtained by the proposed method of UAV’s three-dimensional path planning achieves the expectation.