Backstepping method tracking control for underactuated AUV with unknown dynamics based on action-critic networks based ADP

Abstract

This work investigates trajectory-tacking control problem for underactuated autonomous underwater vehicles (AUV) with unknown dynamics. Due to the unknown dynamics, an action-critic networks based adaptive dynamic programming (ADP) scheme combined with backstepping approach is designed, which can achieve high-level system stability and tracking control accuracy. Firstly, the backstepping approach is introduced into the kinematic model of underactuated AUV and produces a virtual velocity control which is taken as the desired velocity input of the dynamic model of underactuated AUV. Secondly, the error tracking system is constructed according to the dynamic model of underactuated AUV. Thirdly, the critic neural network and the action neural network are employed to transform the trajectory-tracking control problem into optimal control problem based on policy iteration algorithm. At last simulation results are given to verify the effectiveness of the proposed control scheme.

1 Introduction

The motion control of autonomous underwater vehicle(AUV) is still a hot research topic due to its wide applications [1]. Tracking control is one fundamental functionality of motion control for AUV [2, 3]. Many control schemes are developed to solve the trajectory-tracking problem for fully-actuated AUV, such as sliding mode control method [4, 5], adaptive control method [6], observer based adaptive dynamic programming (ADP) [7], backstepping method [8] and dynamic surface control method[9].

However, Some AUV is underactuated and the number of independent control variables of underactuated AUV is less than the number of degrees of freedom (DOF) of underactuated AUV. The underactuated AUV is a complex system [10]. The unknown dynamics could reduce the tracking accuracy and system stability of underactuated AUV, which adds more difficulties in the process of trajectory-tracking control. These difficulties serve as the motivation of this work.

Traditionally, the tracking control problems of underactuated AUV have been solved through a variety of control technologies. The robust control method [11] and backstepping method [12, 13] can deal with high-order nonlinear systems. However, the backstepping method depends on the accurate model of AUV. And the robust control method can cause the chattering phenomenon. The adaptive neural control method does not depend on the accurate model of AUV. The neural network is introduced to solve the trajectory-tracking problem for AUV due to its strong nonlinear fitting ability, strong robustness and strong self-learning ability. The adaptive neural network tracking controller [14] constructed with the radial basic function neural network is developed for underwater vehicle with unknown nonlinear function. The deterministic policy gradient [15] using multi Pseudo Q-learning method is proposed to reduce the overestimation of action-value function for underactuated AUV with unknown dynamics and constrained inputs. The neural networks are employed to approximate the uncertain underactuated vessel dynamics and external disturbances [16]. The radial basis function neural network is employed to deal with the uncertain nonlinear dynamics for the underactuated vessel [17]. The neural network is designed to solve the cooperative path following problem for a fleet of underactuated AUV with uncertain nonlinear dynamics [18]. The neural network is employed to estimate the unknown nonlinear dynamics of AUV [19]. Like the adaptive neural control method, fuzzy control method does not know the accurate model of AUV. However, the the fuzzy control method strongly depend on the prior knowledge. A robust adaptive control scheme based on fully-tuned fuzzy neural network is proposed to solve the trajectory and attitude problem for unmanned underwater vehicle with thruster dynamics and unknown disturbances [20]. Sliding mode method can overcome the influence of unknown disturbance of AUV. However, it will happen chatter in practical applications. Although a number of traditional control schemes have given solutions to the tracking control problems for AUV with different situations, it is necessary to develop more powerful control schemes to meet higher requirements for control performance of AUV. Some researchers have began to focus on two-player zero-sum game, which has received great development in recent years.

However, the mentioned researches usually employed state observer to estimate system state due to the unknown disturbances or uncertainties, which may accumulate the error. Therefore, it is necessary to seek one optimal control scheme for underactuated AUV, which can minimize the performance index function, and also can reduce the effect of unknown dynamics. Unfortunately, there is a few researches that focus on coping with adaptive dynamic programming (ADP) based trajectory-tracking control for underactuated AUV with unknown dynamics. As a powerful optimization tool, ADP scheme has received more attention [21]. ADP scheme has been introduced to compensate for unknown dynamics, such as control input nonlinearities and model uncertainties for nonlinear systems [22, 23]. Policy iteration [24–27] and value iteration [28, 29] are two primary iteration ADP algorithms. The main contributions of this paper can be stated as below:

The virtual velocity control input is designed using the backstepping method, which is taken as the reference velocity input of the dynamic model of underactuated AUV.

The unknown dynamics [21, 30] is considered in this work. Then, the action-critic networks based ADP is employed to transform trajectory-tracking control problem into optimal control problem.

The online policy iteration algorithm and weight update laws are designed. In order to verify the effectiveness of the method proposed in this paper, the simulation results are given with the compared method proposed in [31].

This work is organized as follows. In Section 2, The kinematic model and the dynamic model of underactuated AUV based on the reference model of AUV in [32] are given. In Section 3, ADP optimal controller is designed via backstepping method. In Section 4, two simulation examples are provided. Section 5 gives the conclusions.

2 Problem formulation and mathematical model of AUV

The mathematica model of underactuated AUV is shown in Fig. 1. Two coordinate systems are used that one is the universal frame {O _e  - X _e Y _e Z _e } and the other is the body-fixed frame {O _b  - X _b Y _b Z _b }.

Fig. 1

Underactuated AUV model system.

Table 1

Notation of motion for underactuated AUV

DOF(degree of freedom)	description	Velocity	position and attitude
1	surge	u	x
2	sway	v	y
3	heave	w	z
4	roll	p	φ
5	pitch	q	θ
6	yaw	r	ψ

2.1 Kinematic model of AUV

The kinematic model is given as follows: $\dot{\eta }=J\left(\eta \right)\xi$(1) where J (η) is the coordinate transformation matrix.

2.2 Dynamic model of AUV

The dynamic model of underactuated AUV is established as follows: $M\dot{\xi }+C\left(\xi \right)\xi +D\left(\xi \right)\xi +g\left(\eta \right)+{\tau }_{\varepsilon }=\tau$(2) where M, C (ξ), D (ξ), τ _ɛ , τ and g (η) are the inertia matrix, the Coriolis and centripetal matrix, the hydrodynamic damping matrix, the force vector introduced by the unknown dynamics, the thrust force vector and the gravity and buoyancy forces vector respectively.

The dynamic model (2) can be rewritten as follows: $\dot{\xi }=M^{-1}(-C(\xi )\xi -D(\xi )\xi -g(\eta )-{\tau }_{\varepsilon }+\tau ).$(3)

2.3 Problem formulation

The position error is defined as follows: $e_{\eta }=\eta -{\eta }_d.$(4)

The time derivative of (4) is given as follows: ${\dot{e}}_{\eta }=\dot{\eta }-{\dot{\eta }}_d=J\left(\eta \right)\xi -J\left({\eta }_d\right){\xi }_d.$(5)

The velocity error is defined as follows: $e_{\xi }=\xi -{\xi }_{\mathit{vr}}$(6) where ξ _vr is the virtual velocity control input based on the backstepping approach.

The time derivative of (6) is given as follows: ${\dot{e}}_{\xi }=\dot{\xi }-{\dot{\xi }}_{\mathit{vr}}.$(7)

3 ADP based optimal control design via backstepping method

The block diagram of the proposed control scheme is show in Fig. 2.

Fig. 2

Underactuated AUV controller block diagram.

3.1 Kinematic control via backstepping method

Considering the kinematic model of underactuated AUV, we make e _η  = 0 as t→ ∞. According to (4), (5) and (6), we can get the virtual velocity control input ξ _vr , which is given as follows: ${\xi }_{\mathit{vr}}=J^{-1}(e_{\eta }+{\eta }_d)\left(J\right({\eta }_d){\xi }_d-k_1{e}_{\eta })$(8) where k₁ > 0 and J (e _η  + η _d ) is invertible.

The time derivative of (8) is given as follows: $\begin{array}{cc}{\dot{\xi }}_{\mathit{vr}}\hfill & =J^{-1}(e_{\eta }+{\eta }_d)\left(\dot{J}\right({\eta }_d){\xi }_d+J({\eta }_d){\dot{\xi }}_d\hfill \\ \hfill & -\dot{J}(e_{\eta }+{\eta }_d){\xi }_{\mathit{vr}}-k_1{\dot{e}}_{\eta })\hfill \\ \hfill & =J^{-1}(e_{\eta }+{\eta }_d)\left(\dot{J}\right({\eta }_d){\xi }_d+J({\eta }_d){\dot{\xi }}_d\hfill \\ \hfill & -\dot{J}(e_{\eta }+{\eta }_d)J^{-1}(e_{\eta }+{\eta }_d)\hfill \\ \hfill & \left(J\right({\eta }_d){\xi }_d-k_1{e}_{\eta })-k_1{\dot{e}}_{\eta })\hfill \\ \hfill & ={\Theta }_1+k_1{J}^{-1}(e_{\eta }+{\eta }_d)\dot{J}(e_{\eta }+{\eta }_d)J^{-1}(e_{\eta }\hfill \\ \hfill & +{\eta }_d)e_{\eta }-k_1{J}^{-1}(e_{\eta }+{\eta }_d){\dot{e}}_{\eta }\hfill \end{array}$(9) where ${\Theta }_1=J^{-1}(e_{\eta }+{\eta }_d)\left(\dot{J}\right({\eta }_d){\xi }_d+J({\eta }_d){\dot{\xi }}_d-\dot{J}(e_{\eta }+{\eta }_d\left)J^{-1}\right(e_{\eta }+{\eta }_d\left)J\right({\eta }_d\left){\xi }_d\right)$ .

Lemma 1. Under the proposed virtual control input ξ _vr and the kinematic model of underactuated AUV (1), e _η is UUB.

Proof 1. Choose the Lyapunv function as follows: $V_1=\frac{1}{2}{e}_{\eta }^T{e}_{\eta }.$(10)

The derivative of (10) is calculated as follows: $\begin{array}{cc}\dot{V_1}\hfill & =e_{\eta }^T(\dot{\eta }-\dot{{\eta }_d})\hfill \\ \hfill & =e_{\eta }^T\left(J\right(\eta )\xi -J({\eta }_d\left){\xi }_d\right)\hfill \\ \hfill & =e_{\eta }^T\left(J\right(\eta ){\xi }_{\mathit{vr}}-J({\eta }_d\left){\xi }_d\right)\hfill \\ \hfill & =e_{\eta }^T\left(J\right(\eta \left)J^{-1}\right(\eta \left)\right(J\left({\eta }_d\right){\xi }_d-k_1{e}_{\eta })\hfill \\ \hfill & -J\left({\eta }_d\right){\xi }_d)\hfill \\ \hfill & =e_{\eta }^T(-k_1{e}_{\eta })\hfill \\ \hfill & =-k_1{e}_{\eta }^T{e}_{\eta }\hfill \\ \hfill & =-k_1\parallel e_{\eta }{\parallel }^2\hfill \\ \hfill & \le 0.\hfill \end{array}$(11)

From (11), the time derivative of V₁ is negative semi-definite. It can be concluded that V₁ is bounded and e _η is UUB. This completes the proof.

According to (4)-(9), the dynamic model (2) can be transformed as follows: $\begin{array}{cc}\tau \hfill & =M{\dot{e}}_{\xi }-k_1{\mathit{MJ}}^{-1}(e_{\eta }+{\eta }_d){\dot{e}}_{\eta }+\left(C\right(e_{\xi }\hfill \\ \hfill & +{\xi }_{\mathit{vr}})+D(e_{\xi }+{\xi }_{\mathit{vr}}\left)\right)e_{\xi }+k_1\left({\mathit{MJ}}^{-1}\right(e_{\eta }\hfill \\ \hfill & +{\eta }_d\left)\dot{J}\right(e_{\eta }+{\eta }_d\left)J^{-1}\right(e_{\eta }+{\eta }_d)-(C(e_{\xi }\hfill \\ \hfill & +{\xi }_{\mathit{vr}})+D(e_{\xi }+{\xi }_{\mathit{vr}}\left)\right)J^{-1}(e_{\eta }+{\eta }_d))e_{\eta }\hfill \\ \hfill & +{\Theta }_2+{\tau }_{\varepsilon }\hfill \end{array}$(12) where Θ₂ = MΘ₁ + (C (e _ξ  + ξ _vr ) + D (e _ξ  + ξ _vr )) J^-1 (e _η  + η _d ) J (η _d ) ξ _d  + g (e _η  + η _d ).

Let Λ = [e _ξ  e _η ]  ^T , we can get the error tracking system as follows: $A\dot{\Lambda }+B\Lambda +E=\Gamma$(13) where $A=\left[\begin{array}{cc}\hfill M\hfill & \hfill -k_1{\mathit{MJ}}^{-1}(e_{\eta }+{\eta }_d)\hfill \\ \hfill 0^{6\times 6}\hfill & \hfill I^{6\times 6}\hfill \end{array}\right]$ ; $\Gamma =\left[\begin{array}{c}\hfill {\tau }_e\hfill \\ \hfill 0^{6\times 1}\hfill \end{array}\right]$ ; ${\Theta }_3={\Theta }_2+{\tau }_{\varepsilon }-M{\dot{\xi }}_d-C\left({\xi }_d\right){\xi }_d-D\left({\xi }_d\right){\xi }_d-g\left({\eta }_d\right)$ ; $E=\left[\begin{array}{c}\hfill {\Theta }_3\hfill \\ \hfill J\left({\eta }_d\right){\xi }_d-J\left(\eta \right)J^{-1}\left(\eta \right)J\left({\eta }_d\right){\xi }_d\hfill \end{array}\right]$ ; $B=\left[\begin{array}{cc}\hfill C\left(\xi \right)+D\left(\xi \right)\hfill & \hfill {\Theta }_4\hfill \\ \hfill -J\left(\eta \right)\hfill & \hfill k_{1}J\left(\eta \right)J^{-1}\left(\mathit{eta}\right)\hfill \end{array}\right]$ ; ${\Theta }_4=k_1\left({\mathit{MJ}}^{-1}\right(e_{\eta }+{\eta }_d\left)\dot{J}\right(e_{\eta }+{\eta }_d\left)J^{-1}\right(e_{\eta }+{\eta }_d)-(C(e_{\xi }+{\xi }_{\mathit{vr}})+D(e_{\xi }+{\xi }_{\mathit{vr}})\left)J^{-1}\right(e_{\eta }+{\eta }_d\left)\right)$ ; τ _e  = τ - τ _d and ${\tau }_d=M{\dot{\xi }}_d+C\left({\xi }_d\right){\xi }_d+D\left({\xi }_d\right){\xi }_d+g\left({\eta }_d\right)$ .

3.2 ADP based optimal control design

The performance index function is defined as follows: $V_2\left(\Lambda \right(t),\Gamma )={\int }_t^{\infty }{e}^{\gamma (t-\sigma )}U\left(\Lambda \right(\sigma ),\Gamma (\sigma \left)\right)d\sigma$(14) where γ is the discount factor and 0 ≤ γ < 1; U (Λ (σ) , Γ (σ)) is the utility function and U (Λ (σ) , Γ (σ)) = Λ ^T  (σ) QΛ (σ) + Γ ^T  (σ) RΓ (σ); Q and R are the positive definite matrices.

If the performance index function (14) is continuously differentiable, then the Hamilton-Jacobi-Bellman (HJB) function can be derived as follows: $\begin{array}{cc}H(\Lambda ,\Gamma ,\nabla V_2(\Lambda \left)\right)\hfill & =U(\Lambda ,\Gamma )+(\nabla V_2(\Lambda ){)}^T\hfill \\ \hfill & \times (-A^{-1}(B\Lambda +E\hfill \\ \hfill & -\Gamma \left)\right)-\gamma V_2\left(\Lambda \right)\hfill \\ \hfill & =0\hfill \end{array}$(15) where $\nabla V_2\left(\Lambda \right)=\nabla V_2\left(\Lambda \right(t),\Gamma )=\frac{\partial V_2}{\partial \Lambda }$ and V₂ (0) =0.

According to the Bellman’s optimality principle, the optimal performance index function can be represented as follows: $V_2^{*}\left(\Lambda \right)=\underset{\Gamma \in {\Omega }_{\Gamma }}{min}{\int }_t^{\infty }{e}^{\gamma (t-\sigma )}U\left(\Lambda \right(\sigma ),\Gamma (\sigma \left)\right)d\sigma .$(16)

The optimal performance index function (16) satisfies the HJB function (15) as follows: $\underset{\Gamma }{min}H(\Lambda ,\Gamma ,\nabla V_2^{*}(\Lambda \left)\right)=0.$(17)

The optimal control law is ${\Gamma }^{*}\left(\Lambda \right)=-\frac{1}{2}{R}^{-1}(A^{-1}{)}^T\nabla V_2^{*}(\Lambda ).$(18)

Considering the optimal law (18), the HJB function can be transformed as follows: $\begin{array}{cc}0\hfill & =U(\Lambda ,{\Gamma }^{*})+(\nabla V_2^{*}(\Lambda \left){)}^T\right(-A^{-1}(B\Lambda \hfill \\ \hfill & +E-{\Gamma }^{*}\left)\right)-\gamma V_2^{*}\left(\Lambda \right)\hfill \\ \hfill & ={\Lambda }^{T}Q\Lambda -\frac{1}{4}(\nabla V_2^{*}(\Lambda ){)}^T{A}^{-1}{R}^{-1}\hfill \\ \hfill & \times (A^{-1}{)}^T\nabla V_2^{*}(\Lambda )+(\nabla V_2^{*}\left(\Lambda \right){)}^T\hfill \\ \hfill & \times (-A^{-1}(B\Lambda +E)-\gamma V_2^{*}(\Lambda )\hfill \end{array}$(19) where $V_2^{*}\left(0\right)=0$ .

Remark 1. In (13), Γ is taken as the control law with unknown dynamics. So, an action-critic networks based ADP is employed. The critic network is designed to estimate the optimal performance index function (16) and the action network is designed to estimate the optimal control law (18).

3.3 Critic network design

The critic network is designed to approximate $V_2^{*}\left(\Lambda \right)$ as follows: $V_2^{*}\left(\Lambda \right)=W_1^T{\varpi }_1\left(\Lambda \right)+{\delta }_1\left(\Lambda \right)$(20) where W₁ is the unknown ideal constant weight of critic network; ϖ₁ (Λ) is the activation function vector of critic network; and δ₁ (Λ) is the approximate error of critic network.

The derivative of equation (20) is represented as follows: $\nabla V_2^{*}\left(\Lambda \right)=\nabla {\varpi }_1^T\left(\Lambda \right)W_1+\nabla {\delta }_1\left(\Lambda \right)$(21) where $\nabla {\varpi }_1\left(\Lambda \right)=\frac{\partial {\varpi }_1\left(\Lambda \right)}{\partial \Lambda }$ and $\nabla {\delta }_1\left(\Lambda \right)=\frac{\partial {\delta }_1\left(\Lambda \right)}{\partial \Lambda }$ .

Let ${\hat{W}}_1$ be the approximation of W₁, the approximation of $V_2^{*}\left(\Lambda \right)$ can be represented as follows: ${\hat{V}}_2\left(\Lambda \right)={\hat{W}}_1^T{\varpi }_1\left(\Lambda \right).$(22)

Then, the HJB function can be derived as follows: $\begin{array}{cc}H(\Lambda ,\hat{\Gamma },{\hat{W}}_1)\hfill & ={\hat{W}}_1^T\nabla {\varpi }_1\left(\Lambda \right)(-A^{-1}(B\Lambda \hfill \\ \hfill & +E-\hat{\Gamma }\left)\right)+U(\Lambda ,\hat{\Gamma })\hfill \\ \hfill & -\gamma {\hat{W}}_1^T{\varpi }_1\left(\Lambda \right)\hfill \\ \hfill & =e_{w_1}.\hfill \end{array}$(23)

The square residual error $e_1\left({\hat{W}}_1\right)$ is defined as follows: $e_1\left({\hat{W}}_1\right)=\frac{1}{2}{e}_{w_1}^T{e}_{w_1}.$(24)

Given any admissible control law $\hat{\Gamma }$ , it is desired to select ${\hat{W}}_1$ to minimize $e_1\left({\hat{W}}_1\right)$ . The weight update law based on the gradient descent algorithm is given as follows: ${\dot{\stackrel{ˆ}{W}}}_1=-{\alpha }_1{\varrho }_1({\varrho }_1^T{\hat{W}}_1+{\Lambda }^{T}Q\Lambda +{\hat{\Gamma }}^{T}R\hat{\Gamma })$(25) where α₁ is the adaptive gain of critic network and α₁ > 0; ${\varrho }_1=\frac{\varrho }{{\varrho }^T\varrho +1}$ and $\varrho =\nabla {\varpi }_1(-A^{-1}(B\Lambda +E-\hat{\Gamma })-\gamma {\varpi }_1(\Lambda )$ .

According to the definition of ϱ₁, there exist positive constants ϱ_1M > 1 and ϱ_1m > 0 such that ϱ_1m ≤ ∥ ϱ₁ ∥ ≤ ϱ_1M.

(25) can be transformed as follows: $\begin{array}{cc}{\dot{\tilde{W}}}_1\hfill & ={\dot{\stackrel{ˆ}{W}}}_1-{\dot{W}}_1\hfill \\ \hfill & =-{\alpha }_1{\varrho }_1({\varrho }_1^T{\tilde{W}}_1-\nabla {\delta }_1(\Lambda \left)\right(-A^{-1}(B\Lambda \hfill \\ \hfill & +E-\hat{\Gamma }\left)\right))\hfill \\ \hfill & =-{\alpha }_1{\varrho }_1({\varrho }_1^T{\tilde{W}}_1+{\varepsilon }_{w1})\hfill \end{array}$(26) where ${\varepsilon }_{w1}=-\nabla {\delta }_1\left(\Lambda \right)(-A^{-1}(B\Lambda +E-\hat{\Gamma }\left)\right)$ .

3.4 Action network design

The action network is designed to approximate Γ^* as follows: ${\Gamma }^{*}=W_2^T{\varpi }_2\left(\Lambda \right)+{\delta }_2\left(\Lambda \right)$(27) where W₂ is the unknown ideal constant weight of action network; ϖ₂ (Λ) is the activation function vector of action network; δ₂ (Λ) is the approximation error of action network.

Let ${\hat{W}}_2$ be the approximation of W₂, the actual output can be expressed as follows: $\hat{\Gamma }={\hat{W}}_2^T{\varpi }_2\left(\Lambda \right).$(28)

According to (18) and (28), the feedback error is defined as follows: $e_{w_2}={\hat{W}}_2^T{\varpi }_2+\frac{1}{2}{R}^{-1}(A^{-1}{)}^T\nabla {\varpi }_1^T(\Lambda ){\hat{W}}_1.$(29)

The square residual error $e_2\left({\hat{W}}_2\right)$ is defined as follows: $e_2\left({\hat{W}}_2\right)=\frac{1}{2}{e}_{w_2}^T{e}_{w_2}.$(30)

It is desired to select ${\hat{W}}_2$ to minimize the objective function. The weight update law based on the gradient descent algorithm is given as follows: $\begin{array}{cc}{\dot{\stackrel{ˆ}{W}}}_2\hfill & =-{\alpha }_2{\varpi }_2\left(\Lambda \right)\left({\varpi }_2^T\right(\Lambda ){\hat{W}}_2\hfill \\ \hfill & +\frac{1}{2}{R}^{-1}{A}^{-1}\nabla {\varpi }_1^T\left(\Lambda \right){\hat{W}}_1{)}^T\hfill \end{array}$(31) where α₂ is the adaptive gain of action network and α₂ > 0.

The weight update law (31) can be rewritten as follows: $\begin{array}{cc}{\dot{\tilde{W}}}_2\hfill & ={\dot{\stackrel{ˆ}{W}}}_2-{\dot{W}}_2\hfill \\ \hfill & =-{\alpha }_2{\varpi }_2\left(\Lambda \right)\left({\varpi }_2^T\right(\Lambda ){\tilde{W}}_2+R^{-1}{A}^{-1}\frac{\nabla {\delta }_1\left(\Lambda \right)}{2}\hfill \\ \hfill & +\frac{1}{2}{R}^{-1}{A}^{-1}\nabla {\varpi }_1^T\left(\Lambda \right){\tilde{W}}_1-{\delta }_2\left(\Lambda \right))\hfill \\ \hfill & =-{\alpha }_2{\varpi }_2\left(\Lambda \right)\left({\varpi }_2\right(\Lambda {)}^T{\tilde{W}}_2+{\varepsilon }_{w2})\hfill \\ \hfill & +\frac{1}{2}{R}^{-1}{A}^{-1}\nabla {\varpi }_1^T\left(\Lambda \right){\tilde{W}}_1\hfill \end{array}$(32) where ${\varepsilon }_{w2}=-\left({\delta }_2\right(\Lambda )-R^{-1}{A}^{-1}\frac{\nabla {\delta }_1\left(\Lambda \right)}{2})$ .

3.5 Stability analysis

According to the optimal control $\hat{\Gamma }$  (28) and the error tracking system (13), we have $A\dot{\Lambda }+B\Lambda +E={\hat{W}}_2^T{\varpi }_2\left(\Lambda \right).$(33)

The error tracking system (33) can be transformed using equation (27) as follows: $A\dot{\Lambda }+B\Lambda +E={\tilde{W}}_2^T{\varpi }_2\left(\Lambda \right)+{\Gamma }^{*}-{\delta }_2\left(\Lambda \right).$(34)

Assumption 1 The unknown ideal constant weights W₁ and W₂ satisfy that ∥W₁ ∥ ≤ W_1M and ∥W₂ ∥ ≤ W_2M respectively. W_1M and W_2M are the positive constants.

Assumption 2 The approximation errors of the networks δ₁ and δ₂ satisfy that δ_1m ≤ ∥ δ₁ ∥ ≤ δ_1M and δ_2m ≤ ∥ δ₂ ∥ ≤ δ_2M respectively. δ_1m, δ_1M, δ_2m and δ_2M are the positive constants.

Assumption 3 The the activation function vectors ϖ₁ and ϖ₂ satisfy that ϖ_1m ≤ ∥ ϖ₁ ∥ ≤ ϖ_1M and ϖ_2m ≤ ∥ ϖ₂ ∥ ≤ ϖ_2M respectively. ϖ_1m, ϖ_1M, ϖ_2m and ϖ_2M are the positive constants.

Assumption 4 The gradient of activation function vector of critic network ∇ϖ₁ satisfies that ${\dot{\varpi }}_{1m}\le \parallel \nabla {\varpi }_1\parallel \le {\dot{\varpi }}_{1M}$ . ${\dot{\varpi }}_{1m}$ and ${\dot{\varpi }}_{1M}$ are the positive constants.

Theorem 5. For the error tracking system (13), Assumptions 3.5-3.5 hold. The optimal cost function and the optimal control law are provided by (16) and (18). The weight updating laws of the critic neural network and the action neural network are given by (25) and (31). Then the tracking error Λ and the weight estimate error ${\tilde{W}}_1$ and ${\tilde{W}}_2$ are asymptotically stable.

Proof 2. Let us choose a Lyapunov function candidate composed of three positive terms $L\left(t\right)=L_1\left(t\right)+L_2\left(t\right)+L_3\left(t\right)$(35) where $L_1\left(t\right)=\frac{1}{2{\alpha }_1}\mathit{tr}\left\{{\tilde{W}}_1^T{\tilde{W}}_1\right\}$ ; $L_2\left(t\right)=\frac{1}{2{\alpha }_2}\mathit{tr}\left\{{\tilde{W}}_2^T{\tilde{W}}_2\right\}$ ; L₃ (t) = Λ ^T Λ + V₂ (Λ, Γ)).

The derivative of the Lyapunov function candidate (35) along the trajectories of the error tracking system (34) is given as follows: $\dot{L}\left(t\right)={\dot{L}}_1\left(t\right)+{\dot{L}}_2\left(t\right)+{\dot{L}}_3\left(t\right).$(36)

According to (26), ${\dot{L}}_1$ can be represented as follows: $\begin{array}{cc}{\dot{L}}_1\left(t\right)\hfill & =\frac{1}{{\alpha }_1}\mathit{tr}\left\{{\tilde{W}}_1^T{\dot{\tilde{W}}}_1\right\}\hfill \\ \hfill & =\frac{1}{{\alpha }_1}\mathit{tr}\left\{{\tilde{W}}_1^T\right(-{\alpha }_1{\varrho }_1({\varrho }_1^T{\tilde{W}}_1+{\varepsilon }_{w1})\left)\right\}\hfill \\ \hfill & =\frac{1}{{\alpha }_1}\mathit{tr}\{-{\alpha }_1({\tilde{W}}_1^T{\varrho }_1{\varrho }_1^T{\tilde{W}}_1+{\tilde{W}}_1^T{\varrho }_1{\varepsilon }_{w1}\left)\right\}\hfill \\ \hfill & \le -\frac{3}{4}\parallel {\varrho }_1{\parallel }^2\parallel {\tilde{W}}_1{\parallel }^2+\parallel {\varepsilon }_{w1}{\parallel }^2\hfill \\ \hfill & \le -\frac{3}{4}{\varrho }_{1m}^2\parallel {\tilde{W}}_1{\parallel }^2+\parallel {\varepsilon }_{w1}{\parallel }^2.\hfill \end{array}$(37)

Hence, ${\dot{L}}_1<0$ if $\parallel {\tilde{W}}_1\parallel >\frac{2\parallel {\varepsilon }_{w1}\parallel }{\sqrt{3}{\varrho }_{1m}}$ .

According to equation (32), ${\dot{L}}_2$ can be represented as follows: $\begin{array}{cc}{\dot{L}}_2\left(t\right)\hfill & =\frac{1}{{\alpha }_2}\mathit{tr}\left\{{\tilde{W}}_2^T{\dot{\tilde{W}}}_2\right\}\hfill \\ \hfill & =\frac{1}{{\alpha }_2}\mathit{tr}\left\{{\tilde{W}}_2^T\right(-{\alpha }_2{\varpi }_2\left(\Lambda \right){\varpi }_2(\Lambda {)}^T{\tilde{W}}_2\hfill \\ \hfill & +\frac{1}{2}{R}^{-1}{A}^{-1}\nabla {\varpi }_1^T\left(\Lambda \right){\tilde{W}}_1+{\varepsilon }_{w2}\left)\right)\}\hfill \\ \hfill & =-\mathit{tr}\left\{{\tilde{W}}_2^T{\varpi }_2\right(\Lambda \left){\varpi }_2^T\right(\Lambda ){\tilde{W}}_2\hfill \\ \hfill & +\frac{1}{2}{\tilde{W}}_2^T{R}^{-1}{A}^{-1}\nabla {\varpi }_1^T\left(\Lambda \right){\tilde{W}}_1\hfill \\ \hfill & +{\tilde{W}}_2^T{\varepsilon }_{w2}\}\hfill \\ \hfill & \le -\parallel {\varpi }_2{\parallel }^2\parallel {\tilde{W}}_2{\parallel }^2+2\parallel {\tilde{W}}_2{\parallel }^2\hfill \\ \hfill & +\parallel R{\parallel }^{-2}\parallel A{\parallel }^{-2}\parallel \nabla {\varpi }_1\left(\Lambda \right){\parallel }^2\parallel {\tilde{W}}_1{\parallel }^2\hfill \\ \hfill & +\frac{1}{4}\parallel {\varepsilon }_{w2}{\parallel }^2\hfill \\ \hfill & \le -({\varpi }_{2m}^2-2)\parallel {\tilde{W}}_2{\parallel }^2+\frac{1}{4}\parallel {\varepsilon }_{w2}{\parallel }^2\hfill \\ \hfill & +{\dot{\varpi }}_{1M}^2\parallel \parallel R{\parallel }^{-2}A{\parallel }^{-2}\parallel {\tilde{W}}_1{\parallel }^2.\hfill \end{array}$(38)

Hence, ${\dot{L}}_2<0$ if $\parallel {\tilde{W}}_2\parallel >\sqrt{\frac{{\Theta }_5}{{\varpi }_{2m}^2-2}}$ , ${\Theta }_5={\dot{\varpi }}_{1M}^2\parallel R{\parallel }^{-2}\parallel \left[M\right(Z\left)\right]{\parallel }^{-2}\parallel {\tilde{W}}_1{\parallel }^2+\frac{1}{4}\parallel {\varepsilon }_{w2}{\parallel }^2$ and ${\varpi }_{2m}>\sqrt{2}$ .

${\dot{L}}_3$ is given as follows: $\begin{array}{cc}{\dot{L}}_3\left(t\right)\hfill & ={\Lambda }^T\dot{\Lambda }-{\Lambda }^{T}Q\Lambda -{\Gamma }^{*T}R{\Gamma }^{*}\hfill \\ \hfill & ={\Lambda }^T\left(A^{-1}\right(-B\Lambda -E+{\tilde{W}}_2^T{\varpi }_2\left(\Lambda \right)\hfill \\ \hfill & +{\Gamma }^{*}-{\delta }_2\left(\Lambda \right))-{\Lambda }^{T}Q\Lambda \hfill \\ \hfill & -{\Gamma }^{*T}R{\Gamma }^{*}\hfill \\ \hfill & =-{\Lambda }^T{A}^{-1}B\Lambda -{\Lambda }^T{A}^{-1}E\hfill \\ \hfill & +{\Lambda }^T{A}^{-1}{\tilde{W}}_2^T{\varpi }_2\left(\Lambda \right)+{\Lambda }^T{A}^{-1}{\Gamma }^{*}\hfill \\ \hfill & -{\Lambda }^T{A}^{-1}{\delta }_2-{\Lambda }^{T}Q\Lambda -{\Gamma }^{*T}R{\Gamma }^{*}\hfill \\ \hfill & \le -\parallel A{\parallel }^{-1}\parallel B\parallel \parallel \Lambda {\parallel }^2+4\parallel \Lambda {\parallel }^2+\frac{1}{4}\parallel A{\parallel }^{-2}\parallel E{\parallel }^2\hfill \\ \hfill & +\frac{1}{4}{\varpi }_{2M}^2\parallel A{\parallel }^{-2}\parallel {\tilde{W}}_2{\parallel }^2+\frac{1}{4}\parallel A{\parallel }^{-2}\parallel {\Gamma }^{*}{\parallel }^2\hfill \\ \hfill & +\frac{1}{4}{\delta }_{2M}^2\parallel A{\parallel }^{-2}-{\lambda }_{\mathit{min}}\left(Q\right)\parallel \Lambda {\parallel }^2\hfill \\ \hfill & -{\lambda }_{\mathit{min}}\left(R\right)\parallel {\Gamma }^{*}{\parallel }^2\hfill \\ \hfill & =-(\parallel A{\parallel }^{-1}\parallel B\parallel +{\lambda }_{\mathit{min}}(Q)-4)\parallel \Lambda {\parallel }^2\hfill \\ \hfill & -\left({\lambda }_{\mathit{min}}\right(R)-\frac{1}{4}\parallel A{\parallel }^{-2})\parallel {\Gamma }^{*}{\parallel }^2+\frac{1}{4}\parallel A{\parallel }^{-2}\parallel E{\parallel }^2\hfill \\ \hfill & +\frac{1}{4}{\varpi }_{2M}^2\parallel A{\parallel }^{-2}\parallel {\tilde{W}}_2{\parallel }^2+\frac{1}{4}{\delta }_{2M}^2\parallel A{\parallel }^{-2}.\hfill \end{array}$(39)

Hence, ${\dot{L}}_3<0$ if λ _min  (Q)>4 - ∥ A ∥ ^-1 ∥ B ∥, ${\lambda }_{\mathit{min}}\left(R\right)>\frac{1}{4}\parallel A{\parallel }^{-2}$ and $\parallel \Lambda \parallel >\sqrt{\frac{\frac{1}{4}\parallel A{\parallel }^{-2}\parallel E{\parallel }^2+{\Theta }_6}{\parallel A{\parallel }^{-1}\parallel B\parallel +{\lambda }_{\mathit{min}}\left(Q\right)-4}}$ , ${\Theta }_6=\frac{1}{4}{\varpi }_{2M}^2\parallel A{\parallel }^{-2}\parallel {\tilde{W}}_2{\parallel }^2+\frac{1}{4}{\delta }_{2M}^2\parallel A{\parallel }^{-2}$ .

Then, we can get $\begin{array}{cc}\dot{L}\left(t\right)\hfill & ={\dot{L}}_1+{\dot{L}}_2+{\dot{L}}_3\hfill \\ \hfill & <0.\hfill \end{array}$(40)

Therefore, it can be concluded that the tracking error Λ and the neural network estimation error ${\tilde{W}}_1$ and ${\tilde{W}}_2$ are UUB. This completes the proof.

4 Simulation

The model parameters values of underactuated AUV is given in [2]. The numerical values of the parameters used in the simulations are given that ${\tau }_d^{\prime }=[500,0,0,100,10{]}^T$ , γ = 0.9, α₁ = 0.01, α₂ = 0.01, and k₁ = 1. In order to verify the effectiveness of the proposed control technique, two simulation examples without unknown dynamics and with unknown dynamics are performed compared with the traditional ADP scheme without backstepping method respectively.

4.1 Example one without unknown dynamics

In this section, the simulation results of the underactuated AUV without unknown dynamics are shown in Figs. 3–9.

Fig. 3

e _ξ with method proposed in this work.

Fig. 4

e _ξ with ADP without backstepping method.

Fig. 5

e _η with method proposed in this work.

Fig. 6

e _η with ADP without backstepping method.

Fig. 7

Tracking of desired velocity.

Fig. 8

Tracking of desired position.

Fig. 9

Tracking of desired trajectory.

Figures 3 and 4 show e _ξ of underactuated AUV. The initial value of e _ξ is [0.25, 0, 0, 0.01, 0.01]  ^T . Figures 5 and 6 show e _η of underactuated AUV. The initial value of e _η is [0.01, 0, 0, - 0.01, - 0.01]  ^T . The convergence time of e _ξ and e _η with the proposed method in this paper is shown in Table 2. From above simulation results, we know that e _ξ and e _η converge to 0 with the proposed method in this paper, and e _ξ and e _η do not to 0 with ADP scheme without backstepping method due to the actuator saturation. In addition, the convergence velocity of e _ξ and e _η is faster.

Table 2

Convergence time of tracking error

Tracking error	The proposed method(s)
errorofu /e ξ  (1)	12
errorofv /e ξ  (2)	27
errorofw /e ξ  (3)	38
errorofq /e ξ  (4)	37
errorofr /e ξ  (5)	26
errorofx /e η  (1)	19
errorofy /e η  (2)	29
errorofz /e η  (3)	36
errorofθ /e η  (4)	25
errorofψ /e η  (5)	23

The tracking of desired velocity of underactuated AUV with two different control schemes is given in Figs. 7 and 8 shows the tracking of desired position of underactuated AUV with two different control schemes. Figure 9 illustrates the spatial trajectories of unnderactuated AUV with different control schemes. From above simulation results, the proposed method in this work has better trajectory-trackingaccuracy.

4.2 Example two with unknown dynamics

In this section the unknown dynamics is considered.

Figures 10 and 11 show e _ξ of underactuated AUV. The initial value of e _ξ is [0.25, 0, 0, 0.01, 0.01]  ^T . Figures 12 and 13 show e _η of underactuated AUV. The initial value of e _η is [0.01, 0, 0, - 0.01, - 0.01]  ^T . From Figs. 10 and 11, we can see that e _ξ and e _η converge to 0 with the method proposed in this paper. Due to the unknown dynamics, there exists small bounded chatter in the process of convergence. The convergence time of tracking error with the proposed method in this paper is shown in Table 3. Compared with example one, the convergence velocity is a litter slower due to the unknown dynamics. The time response of the underactuated AUV clearly shows that the proposed control method guarantees a higher-level performance.

Fig. 10

e _ξ with method proposed in this work.

Fig. 11

e _ξ with ADP without bacstepping method.

Fig. 12

e _η with method proposed in this paper.

Fig. 13

e _η with ADP without backstepping method.

Table 3

Convergence time of tracking error

Tracking error	The proposed method(s)
errorofu /e ξ  (1)	12
errorofv /e ξ  (2)	20
errorofw /e ξ  (3)	40
errorofq /e ξ  (4)	40
errorofr /e ξ  (5)	40
errorofx /e η  (1)	19
errorofy /e η  (2)	30
errorofz /e η  (3)	52
errorofθ /e η  (4)	55
errorofψ /e η  (5)	50

The tracking of desired velocity of underactuated AUV with two different control schemes is given in Fig. 14. Figure 15 shows the tracking of desired position of underactuated AUV with two different control schemes. Figure 16 illustrates the spatial trajectories of unnderactuated AUV with different control schems. From above simulation results, the proposed method in this work has better trajectory-tracking accuracy.

Fig. 14

Tracking of desired velocity.

Fig. 15

Tracking of desired position.

Fig. 16

Tracking of desired trajectory.

5 Conclusions

The stability of error tracking system (34) is guaranteed based on the Lyapunov stability theorem. The simulation results have shown excellent convergence of the error tracking systems (34) compared with the ADP scheme without backstepping method. The proposed control scheme achieves good tracking performance.

At the same time, the proposed method lacks the consideration of uncertainties limitations. Future researches will concentrate on improving tracking accuracy and stability for cooperative tracking control problems of multiple AUVs with uncertainties. In addition, deep reinforcement learning will be taken into account in future study.

Declarations

Funding:There is no funding to support this work.

Conflicts of interest:The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this manuscript.

Authors’ contributions:G. Che designs the control method, does the simulation experiments and writes the manuscript and Z. Yu designs the control method and analyzes the stability of system.