Training a model-free reinforcement learning controller for a 3-degree-of-freedom helicopter under multiple constraints

Abstract

The purpose of the article is to design data-driven attitude controllers for a 3-degree-of-freedom experimental helicopter under multiple constraints. Controllers were updated by utilizing the reinforcement learning technique. The 3-degree-of-freedom helicopter platform is an approximation to a practical helicopter attitude control system, which includes realistic features such as complicated dynamics, coupling and uncertainties. The method in this paper first describes the training environment, which consists of user-defined constraints and performance expectations by using a reward function module. Then, actor–critic-based controllers were designed for helicopter elevation and pitch axis. Next, the policy gradient method, which is an important branch of the reinforcement learning algorithms, is utilized to train the networks and optimize controllers. Finally, from experimental results acquired by the 3-degree-of-freedom helicopter platform, the advantages of the proposed method are illustrated by satisfying multiple control constraints.

Introduction

In the unmanned aerial system field, the helicopter control problems have attracted much attention because of their wide applications and scientific significance. Difficulties in designing controllers for helicopters are due to their particular features, such as external disturbance forces, model uncertainties, nonlinear dynamic conditions and coupling axis problems. In the past few decades, abundant papers have been reported about the controller design for the helicopter, see for example, sliding mode controllers,^1,2 robust controller, $H_{\infty }$ controller^3,4 and $L_2$ controller. ⁵ However, the experiments for the practical helicopter tests are time-consuming and high cost, and then few experiment results have been reported.

In order to avoid the difficulties of the practical helicopter experiments, we aim to find a simplified model to corroborate the theoretical application. The helicopter platform in the study by Zheng and Zhong ⁶ provided a simplified model, which consisted of strong coupling characteristic and complicated, nonlinear dynamics. Also, this model utilized the Simulink for controller design, and then it compiled the codes for the practical experiments. Based on this model, a large amount of research papers have been published so far. Rios et al. ⁷ utilized the sliding-mode observation method. The robust controller design methods were brought out by Liu et al. ⁸ Also, Kutay et al. ⁹ used the adaptive method to design the pitch axis output feedback controller within this platform. Rosales dealt in three axes the set-point regulation problem. Li et al. ¹⁰ brought out robust nonlinear controller, which is based on the robust integral of the error, to solve the attitude track problems, both set-point tracking and sine-signal tracking. But, all these controllers are designed by manually adjusting parameters. Also, seldom controllers can deal with control problems under multiple constraints. In this paper, authors want to design controllers for the helicopter platform under multiple constraints only with its input data and output data, which also means model-free controllers.

Among the machine learning method, reinforcement learning (RL) performs well when dealing with serial decision-making problems and control strategies. Sutton and Barto ¹¹ based on the past decade research introduced the RL in their study. One of the fundamental theories is the Bellman equation, which is utilized to optimize value function iterations or policy iterations. Inspired by this, the researchers in control theory field improved the adaptive dynamic programming (ADP) by RL. By using the reward in “Linear Quadratic Regulator (LQR)” form, they used the Hamilton–Jacobi–Bellman (HJB) equation to deal with controller design problems for linear systems,^12–16 discrete nonlinear system,^17–20 complex-valued nonlinear systems, ²¹ nonlinear switched systems, ²² multiagent systems^23–25 and so on. Also, according to ADP, Kiumarsi et al., ²⁶ Luo et al., ²⁷ and Modares et al. ²⁸ brought out $H_{\infty }$ controllers and robust controllers. However, few papers provided practical results for the simulations. Based on the learning theory in the study by Sutton and Barto, ¹¹ researchers applied the deep RL to train agents play simple computer games. ²⁹ Also, they used the Monte Carlo Tree search method to train the intelligence play Go against human, which is famous in the world. ³⁰ However, these methods are only useful for low-dimension discrete systems. Continuous systems bring difficulties for agents to choose actions because the number of the actions can seem to be infinity. Then, they utilized the policy gradient (PG) method and deep network to solve these problems. ³¹

Methods such as ADP and Q-learning are the value-based learning methods. These algorithms alternate actions by estimating the value function or Q-function and then improve the policy. Another type of RL algorithm is the PG method. It improves the policy by an estimator of the gradient of the cost function, which is calculated through the data. Abundant research studies about PG methods have been reported so far, see for example the studies by Peters and Schaal ³² and Luo et al. ³³ But, few of them behaved well in practical plants, ³⁴ because it is difficult for us to choose an appropriate step size. Too short means slow learning rates and too long results in divergence. Schulman et al. ³⁵ solved this problem by giving a limitation for the choice of the step size.

The aim of the article is to design RL-based controllers for the helicopter under multiple constraints. In the practical systems, there exist many constraints, such as the limiting control inputs. Also, researchers want to add some constraints for the performance such as no overshoot. In this article, we use the reward functions to describe these constraints and construct the learning environment. An actor–critic-based controller is brought out for the helicopter experiment plant, and we utilize one type of the PG methods to optimize the controllers. Finally, the table-mount helicopter platform is used to prove the efficiency of the proposed algorithm under different constraint conditions.

Problem formulation

The helicopter model used in this article is the table-mount helicopter, which is shown in Figure 1. The helicopter consists of two parts: a rectangular body frame and two propeller assemblies. The arm in this model allows the elevation and the travel motions of the helicopter with the utilization of a 2-degree-of-freedom (DOF) instrumented joint. The helicopter will elevate when motors are both driven with two positive voltages. If the voltage on the front motor is greater than that on the back motor, the positive pitch movement is generated. The thrust vectors of the body pitches can result in the travel motion. Angle position information can be directly measured by encoders mounted through the middle part.

Figure 1.

Quanser table-mount helicopter platform.

Based on the abovementioned simplified structure of the helicopter model, the control method designed in the article aims to track the reference signals of elevation and pitch axes. Then, we can formulate elevation and pitch motions of the helicopter as follows.

• Elevation motion

$$\begin{array}{l}{J}_e\stackrel{··}{\alpha }=K_f{l}_{\alpha }{V}_s\cos \left(\beta \right)-mg{l}_{\alpha }\cos \left(\alpha \right)\hfill \\ V_s=V_f+V_b\hfill \end{array}$$

(1)

• Pitch motion

$$\begin{array}{c}{J}_p\stackrel{··}{\beta }=K_f{l}_{\beta }{V}_d,V_d=V_f-V_b\hfill \end{array}$$

(2)

where parameter symbols are listed in Table 1.

Table 1.

Symbols for equations (1) and (2).

Symbol	Instruction
$\alpha$	Elevation angle
$\beta$	Pitch angle
$J_e$	Inertia moment of elevation axis
$J_p$	Inertia moment of pitch axis
$l_{\alpha }$	Distance between elevation axis and helicopter
$l_{\beta }$	Distance between pitch axis and helicopter
$m$	Mass of the helicopter
$g$	The gravitational acceleration constant
$K_f$	The force constant of the motor
$V_f,V_b$	Voltages for front motor and back motor
$V_s,V_d$	Control outputs

Define

$$e_e\left(t\right)=r_e\left(t\right)-\alpha \left(t\right),e_p\left(t\right)=r_p\left(t\right)-\beta \left(t\right)$$

where $r_e(t)$ and $r_p(t)$ are the target signals of elevation and pitch axes. $e_e(t)$ and $e_p(t)$ are tracking errors for elevation and pitch motions.

The aim of this paper is to design a controller that can track the signal and satisfy multiple performance constraints. Then, we define these special constraints as $c_1,c_2,\dots$ .

Remark 1

Here, we present some examples for the special constraints $c_1,c_2,\dots$ .

• $c_1$ : the track signal we expect can have small overshoot, even no overshoot.

• $c_2$ : the controller we design can have $H_{\infty }$ performance, which means $\underset{0}{\overset{\infty }{\int }}e(t)\mathrm{dt}<ϵ\underset{0}{\overset{\infty }{\int }}\omega (t)\mathrm{dt}$ . Here, $ϵ$ is a small positive scalar.

• $c_3$ : some variables of the signals have strict limitation, and it is not convenient to directly use a saturation module.

• $c_4$ : the controller can obtain better $H_{\infty }$ performance in a finite frequency range, otherwise, the controller can satisfy finite frequency $H_{\infty }$ performance.

In conclusion, our control objective is to design two model-free controllers for elevation axis and pitch axis such that $e_e(t)\to 0$ , $e_p(t)\to 0$ as $t\to \infty$ . Simultaneously, the closed-loop system can satisfy the special constraints $c_1,c_2,\dots$ . The controllers are designed in an actor–critic structure. The control outputs are $V_s$ and $V_d$ , and then we convert them into $V_f$ and $V_b$ , which are applied on the helicopter experiment.

Preliminary on RL

Policy iteration is an important part of the RL. It is widely known that the Q-function in RL is described as follows

$$\begin{array}{cc}\multicolumn{1}{c}{Q^{{\pi }_{\mathrm{new}}}(s,a)}& =(1-\varrho )Q^{\pi }(s,a)\hfill \\ \multicolumn{1}{c}{}& +\tau (r+\epsilon \underset{a}{\max }{Q}^{\pi }(s\prime ,a))\hfill \end{array}$$

(3)

where $r$ is the reward, $\epsilon$ is the discounted factor and $\rho$ is the learning rate. The policy $\pi$ updates according to

$${\pi }^{*}=\underset{\pi }{\arg \max }{Q}^{\pi }(s,a)$$

(4)

This type of Q-function brings many shortcomings when it is applied to solve the continuous system problem. The action bound and the action amount will be noted as infinity. Then, it is difficult to make a choice, which can bring maximal reward. However, another branch, PG method, can solve such problems.

PG method

In the PG method, ³² we directly parameterize policy $\pi$ with parameters $\theta$ and minimize the cost function as

$$J(\theta )={\mathbb{E}}_{{\rho }_{\theta }}\left[\begin{array}{c}\sum _{t=0}^{\infty }{\gamma }^{t}r(s_t,a_t)\hfill \end{array}\right]$$

where ${\rho }_{\theta }=p(s_0)\pi (a_0|s_0)p(s_1|s_0,a_0)\cdots$ . From the PG theorem, we can obtain its gradient as follows

$${\nabla }_{\theta }J=\mathbb{E}\left[\begin{array}{c}\sum _t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)(R_t-b_t)\hfill \end{array}\right]$$

(5)

where $R_t=\sum _{t\prime =t}{\gamma }^{t\prime -t}r(s_{t\prime },a_{t\prime })$ and $b_t$ is a baseline. We often choose $b_t=V_{\theta }(s_t)={\mathbb{E}}_{\theta }[R_t|s_t]$ . Define the advantage function as $A_{\theta }(s_t,a_t)={\mathbb{E}}_{\theta }[R_t|s_t,a_t]-V_{\theta }(s_t)$ . Then, we have the commonly used gradient estimator

$$g={\mathbb{E}}_t\left[{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)\left(A_{\theta }\left(t\right)\right)\right]$$

(6)

Then, we can optimize the policy by a stochastic gradient descent optimizer to obtain a policy, which can maximize the sum of the reward.

Trust region policy optimization

When we apply the PG method into the practical utilization, it is hard to choose the step size for the optimization. The policy will not converge when our step size is quite large. Also, the optimization process is exceedingly time-consuming when it has small step size.

Schulman et al. ³⁵ brought out the TRPO method to solve this problem by defining a trust region for the step size. In each iteration, TRPO transfers the current parameters $\theta$ to ${\theta }_{\mathrm{old}}$ . Then, it optimizes the policy according to

$$J_{\mathrm{TRPO}}\left({\theta }_{\mathrm{new}}\right)={\mathbb{E}}_t\left[\frac{{\pi }_{{\theta }_{\mathrm{new}}}(a_t|s_t)}{{\pi }_{{\theta }_{\mathrm{old}}}(a_t|s_t)}{A}_{{\theta }_{\mathrm{old}}}\left(t\right)\right]$$

(7)

TRPO sets a constraint for the step size of the policy update. By using Kullback–Leibler (KL) divergence, the constraint can be expressed as $\mathrm{KL}[{\pi }_{{\theta }_{\mathrm{old}}}|{\pi }_{{\theta }_{\mathrm{new}}}]<\delta$ .

Main result

In this section, we will introduce the structure of the table-mount helicopter control system and the controller design method with the consideration under multiple constraints.

The structure of the controllers and the physical plant is shown in Figure 2. From the figure we have two main modules, reward function module and RL controller. Reward function module provides reward for the controller according to multiple constraints. The algorithm updates controllers with the utilization of reward and state signals based on RL. After optimization, the controllers can efficiently track the reference signal and satisfy multiple constraints.

Figure 2.

Helicopter control system.

Reward function module

In the history of the RL, reward plays an important role as one of four basic elements. We set reward rules for the problem and with the algorithm, the agent obtains more and more rewards after each episode. In other words, we utilize reward functions to construct the performance environment, which helps us train the policy. In this paper, the reward we use is continuous with states varying.

From Figure 2, one can see that the total reward $r_t$ consists of automated reward $r_a$ and special reward $r_s$ . Define

$$r_t=r_a+r_s,r_s=\sum _{i=1}^n{w}_i{r}_{\mathrm{ci}}$$

(8)

where $n$ is the amount of the constraints, $r_{\mathrm{ci}}$ is the reward for the $i_{\mathrm{th}}$ constraint and $w_i$ is the weight value for $r_{\mathrm{ci}}$ . Automated reward $r_a$ describes the main task for the controller optimization. It may be different in different problems and in this track problem, we set the value of $r_a$ at time $t$ as

$$r_a\left(t\right)=-\left|\mathrm{Qe}\left(t\right)\right|$$

(9)

where $e(t)$ is the reference signal and $Q$ is parameter matrix we give before training. Then, we give definitions for special reward $r_s$

$$w_i=\{\begin{array}{cc}1,\hfill & r_{\mathrm{ci}}<\underset{¯}{r_{\mathrm{ci}}}\hfill \\ \multicolumn{1}{c}{{\sigma }_i.}& r_{\mathrm{ci}}\ge \underset{¯}{r_{\mathrm{ci}}}\hfill \end{array}$$

(10)

where $\underset{¯}{r_{\mathrm{ci}}}$ denotes the lower robust reward bound for the constraint $c_i$ , $\sigma$ is a sufficient small positive rational number, which is decided by users and in this paper ${\sigma }_i=0.1,i=1,2,\dots ,n$ .

Remark 2

Compared with main target, constraint $c_i$ are described by $r_{\mathrm{ci}}$ , and we suppose each constraint has a tolerable range. We should strictly satisfy the constraint out of range and relatively pay more attention to the main target in the range. Then, the bound for the range is the robust reward bound, which is $[\underset{¯}{r_{\mathrm{ci}}},0]$ . Correspondingly, parameter ${\sigma }_i$ should be smaller when the main task of the control problem is more important.

The $i_{\mathrm{th}}$ constraint reward $r_{\mathrm{ci}}$ is different for different types of constraints. Here, we provide several types of $r_{\mathrm{ci}}$ for reference, which will be used in our works. As mentioned in Remark 1, we design the following rewards for no overshoot condition $c_1$ , $H_{\infty }$ performance condition $c_2$ and strict limitation condition $c_3$ .

$t_{\exp }$ is the response time that user reasonably expect. $e_s(t)$ is the error signal and $e_m$ is the maximum tolerable value of $e_s(t)$ during the training. Let $\phi =0.5|e_m|$ .

$c_1$ : suppose that we want to track signal $s_c(t)$ and the track response has small overshoot or no overshoot. Then

$$\begin{array}{c}{r}_{\mathrm{ci}}\left(t\right)=\{\begin{array}{cc}-h{\stackrel{·}{e}}_s\left(t\right),\hfill & \mathrm{if}{\stackrel{·}{e}}_s\left(t\right)>{\overline{c}}_1\hfill \\ \multicolumn{1}{c}{0.}& \mathrm{otherwise}\hfill \end{array},\underset{¯}{r_{\mathrm{ci}}}=0\hfill \\ \multicolumn{1}{c}{h={\xi }_1\frac{|e_m|}{{\overline{c}}_1},{\overline{c}}_1=2\frac{\phi }{t_{\exp }}}\end{array}$$

(11)

where ${\xi }_1$ is hyperparameter, and in this paper we take ${\xi }_1=20$ .

$c_2$ : suppose the disturbance input during training is $\omega (t)$ , the acceptable maximum value for $\epsilon$ is $\overline{\epsilon }$ and the error signal is $e_s(t)$ for track problem, then

$$\begin{array}{c}{r}_{\mathrm{ci}}\left(t\right)=-\left|\frac{{\Vert e_s\left(t\right)\Vert }_{\infty }}{{\Vert \omega \left(t\right)\Vert }_{\infty }}\right|,\underset{¯}{r_{\mathrm{ci}}}=\overline{ϵ}\hfill \end{array}$$

(12)

$c_3$ : suppose there is strict limitation for signal $s_u(t)$ , the limitation is $[\underset{¯}{s_u},\overline{s_u}]$ and the error signal is $e_s(t)$ for track problem, then $\underset{¯}{r_{\mathrm{ci}}}=0$ and

$$\begin{array}{c}{r}_{\mathrm{ci}}(t)=(\begin{array}{cc}-\left|e_m\right|-\frac{{\xi }_2\left|e_m\right|}{\left|\overline{s_u}\right|+\left|\underset{¯}{s_u}\right|}(s_u\left(t\right)-\overline{s_u}),\hfill & \mathrm{if}{s}_u\left(t\right)>\overline{s_u}\hfill \\ \multicolumn{1}{c}{-\left|e_m\right|-\frac{{\xi }_2\left|e_m\right|}{\left|\overline{s_u}\right|+\left|\underset{¯}{s_u}\right|}(\left|\underset{¯}{s_u}\right|-s_u\left(t\right)),}& \mathrm{if}{s}_u\left(t\right)<\overline{s_u}\hfill \\ \multicolumn{1}{c}{0}& \mathrm{otherwise}\hfill \end{array}\hfill \end{array}$$

(13)

where ${\xi }_2$ is hyperparameter, and in this paper we take ${\xi }_2=10$ .

Based on the above introduction, we can assume that we utilize rewards $r_{\mathrm{ci}},i=1,2,\dots ,n$ to describe multiple constraints and weights $w_i$ to reduce the influence of the constraints to main task. Then, the total reward $r_t$ can describe our control target.

Remark 3

Here, the reward we give for such three types of constraints is efficient in our simulation. We suggest that in similar constraints, the reward can follow similar structures and hyperparameters ${\xi }_1$ and ${\xi }_2$ should be reconsidered. Another point should be noted that ${\overline{c}}_1$ may not be in the same form as ours and $h$ will not always be ${\xi }_1{t}_{\exp }$ . Obviously, the different values of the parameters can influence the speed or the possibility of the convergence. Therefore, the choice of the parameters should be reasonable and cautious.

Remark 4

In previous studies,^{12–14,17–19,21–23} the authors utilized the ADP method to design controllers and the reward they use can be described as $r(k)=x_k^{\mathrm{T}}P{x}_k+u_k^{\mathrm{T}}R{u}_k$ . This reward is used to construct value function and the authors obtain an optimal controller with the HJB equation. However, the HJB equation cannot easily solve optimize problems with reward shown in this paper as $r_t$ . Thus, we propose a PG method to solve this type of optimize problems with multiple constraints.

Optimization method

From section “Preliminary on RL,” we denote the expected discounted cost under a stochastic policy $\pi$ as

$$J_{\pi }={\mathbb{E}}_{s_0,a_0,\dots }\left[\sum _{t=0}^{\infty }{\gamma }^{t}r\left(s_t\right)\right]$$

(14)

where $a_t=\pi (a_t|s_t)$ and $s_{t+1}=P(s_{t+1}|s_t,a_t)$ . $\gamma$ is the discounted constant and $r(s_t)$ is the reward of $s_t$ . According to equation (8), we have

$$\begin{array}{c}{\mathbb{E}}_{s_0,a_0,\dots }\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_t\left(s_t\right)\right]={\mathbb{E}}_{s_0,a_0,\cdots }\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_a\left(s_t\right)\right]\hfill \\ \multicolumn{1}{c}{+\sum _{t=0}^n{\mathbb{E}}_{s_0,a_0,\dots }\left[\sum _{t=0}^{\infty }{\gamma }^t{w}_i{r}_{\mathrm{ci}}\left(s_t\right)\right]}\end{array}$$

(15)

According to equation (10), in the tolerable range, ${\sigma }_i$ is such small that we have

$$J_{\pi }={\mathbb{E}}_{s_0,a_0,\dots }\left[\sum _{t=0}^{\infty }{\gamma }^{t}r\left(s_t\right)\right]\approx {\mathbb{E}}_{s_0,a_0,\dots }\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_a\left(s_t\right)\right]$$

(16)

In our following method, we set $r(s_t)=r_t(s_t)$ . The definitions of the value function $V_{\pi }$ and state-action value function $Q_{\pi }$ are given as follows

$$\begin{array}{c}{V}_{\pi }\left(s_t\right)={\mathbb{E}}_{a_t,s_{t+1},\dots }\left[\sum _{m=0}^{\infty }{\gamma }^{m}r\left(s_{t+m}\right)\right]\hfill \\ \multicolumn{1}{c}{Q_{\pi }(s_t,a_t)={\mathbb{E}}_{s_{t+1},a_{t+1},\dots }\left[\sum _{m=0}^{\infty }{\gamma }^{m}r\left(s_{t+m}\right)\right]}\end{array}$$

(17)

Then, we define the advantage function $A_{\pi }$ as $A_{\pi }(s_t,a_t)=Q_{\pi }(s_t,a_t)-V_{\pi }(s_t)$ . When we update policy from ${\pi }_{\mathrm{old}}$ to ${\pi }_{\mathrm{new}}$ , there is a useful equation shown as follows, whose proof can be found in the study by Kakade and Langford ³⁶

$$J_{{\pi }_{\mathrm{new}}}=J_{{\pi }_{\mathrm{old}}}+{\mathbb{E}}_{s_0,a_0,\dots }\left[\sum _{t=0}^{\infty }{\gamma }^t{A}_{{\pi }_{\mathrm{old}}}(s_t,a_t)\right]$$

(18)

where $a_t={\pi }_{\mathrm{new}}(a_t|s_t)$ and $s_{t+1}=P(s_{t+1}|s_t,a_t)$ . We denote ${\rho }_{\pi }$ , the discounted visitation frequencies as

$${\rho }_{\pi }\left(s\right)=P(s_0=s)+\gamma P(s_1=s)+{\gamma }^{2}P(s_2=s)+\cdots$$

Then, equation (11) can be reformulated as

$$J_{{\pi }_{\mathrm{new}}}=J_{{\pi }_{\mathrm{old}}}+\sum _s{\rho }_{{\pi }_{\mathrm{new}}}\left(s\right)\sum _a{\pi }_{\mathrm{new}}(a|s)A_{{\pi }_{\mathrm{old}}}(s,a)$$

(19)

One can see that from equation (19), we can obtain that if $\sum _a{\pi }_{\mathrm{new}}(a|s)A_{{\pi }_{\mathrm{old}}}(s,a)<0$ , the policy update process from ${\pi }_{\mathrm{old}}$ to ${\pi }_{\mathrm{new}}$ is monotonic. The expected cost will be less after each step update. When the expected advantage function equals to 0, the cost function will reach the minimum. However, in some conditions, due to the estimation error of $V_{{\pi }_{\mathrm{old}}}(s)$ or too long step size, the expected advantage of some states $s$ is positive, which means $\sum _a{\pi }_{\mathrm{new}}(a|s)A_{\pi }(s,a)>0$ . To solve this problem, a method is utilized, which is the conservative policy iteration, brought out by Kakade and Langford. ³⁶ In Theorem 1 of the paper, ³⁵ we can convert equation (19) into the following equations

$$J_{{\pi }_{\mathrm{new}}}\le L_{{\pi }_{\mathrm{old}}}\left({\pi }_{\mathrm{new}}\right)+\epsilon D_{\mathrm{KL}}^{\max }({\pi }_{\mathrm{old}},{\pi }_{\mathrm{new}})$$

(20)

where

$$L_{{\pi }_{\mathrm{old}}}\left({\pi }_{\mathrm{new}}\right)=J_{{\pi }_{\mathrm{old}}}+\sum _s{\rho }_{{\pi }_{\mathrm{old}}}\left(s\right)\sum _a{\pi }_{\mathrm{new}}(a|s)A_{{\pi }_{\mathrm{old}}}(s,a)$$

$$D_{\mathrm{KL}}^{\max }({\pi }_{\mathrm{old}},{\pi }_{\mathrm{new}})=\underset{s}{\max }{D}_{\mathrm{KL}}({\pi }_{\mathrm{old}}(·|s)\parallel {\pi }_{\mathrm{new}}(·|s))$$

$$\epsilon =\frac{2\kappa \gamma }{{(1-\gamma )}^2},\kappa =\underset{s}{\max }\left|\mathbb{E}\left[A_{{\pi }_{\mathrm{old}}(s,a)}\right]\right|$$

where $D_{\mathrm{KL}}$ denotes KL divergence. We transform the abovementioned equation to a sample-based estimation form. By utilizing $q$ as sampling distribution, for one step with state $s_k$ , we have

$$\sum _s{\rho }_{{\pi }_{\mathrm{old}}}\left(s\right)[\cdots ]=\frac{1}{1-\gamma }{\mathbb{E}}_{s~{\rho }_{{\pi }_{\mathrm{old}}}}[\cdots ]$$

$$\sum _a{\pi }_{\mathrm{new}}(a|s)A_{{\pi }_{\mathrm{old}}}(s,a)={\mathbb{E}}_{a~q}\left[\frac{{\pi }_{\mathrm{new}}(a|s_k)}{q(a|s_k)}{A}_{{\pi }_{\mathrm{old}}}(s_k,a)\right]$$

In the estimation procedure, we simulate the plants with policy ${\pi }_{\mathrm{old}}$ to generate series as $s_0,a_0,s_1,\dots$ . We can obtain that $q(a|s)={\pi }_{\mathrm{old}}(a|s)$ , which results in

$$\begin{array}{c}{L}_{{\pi }_{\mathrm{old}}}\left({\pi }_{\mathrm{new}}\right)=J_{{\pi }_{\mathrm{old}}}+\frac{E(s_k,a_k)}{1-\gamma }\hfill \\ E(s_k,a_k)={\mathbb{E}}_{a~q,s~{\rho }_{{\pi }_{\mathrm{old}}}}\left[\begin{array}{c}\frac{{\pi }_{\mathrm{new}}(a_k|s_k)}{q(a_k|s_k)}{A}_{{\pi }_{\mathrm{old}}}(s_k,a_k)\hfill \end{array}\right]\hfill \end{array}$$

(21)

From equation (13), we can find that the penalty parameter $\epsilon$ is so large that the KL divergence is short, which means the step size will be very small. Then, we optimize $L_{{\pi }_{\mathrm{old}}}({\pi }_{\mathrm{new}})$ and give constraints for $\epsilon D_{\mathrm{KL}}^{\max }({\pi }_{\mathrm{old}},{\pi }_{\mathrm{new}})$ . Here, we do not care about the determined part in the optimization process, so minimizing $L_{{\pi }_{\mathrm{old}}}({\pi }_{\mathrm{new}})$ is equal to minimizing $E(s_k,a_k)$ .

In this study, we usually choose the reward $r(s_k)$ not more than zero. In other words, the cost is not more than zero. At the same time, we parameterize policy ${\pi }_{\theta }(a|s)$ with parameter vector $\theta$ . Overload our equations and our objective can be described as

$$\begin{array}{c}\underset{\theta }{\mathrm{maximize}}\mathbb{E}\left[\begin{array}{c}\mathcal{F}\left(\theta \right)A_{{\theta }_{\mathrm{old}}}(s_k,a_k)\hfill \end{array}\right]\hfill \\ \multicolumn{1}{c}{\mathrm{subject}\mathrm{to}{D}_{\mathrm{KL}}^{\max }({\theta }_{\mathrm{old}},\theta )<\delta }\\ \multicolumn{1}{c}{\mathcal{F}(\theta )=\frac{{\pi }_{{\theta }_{\mathrm{new}}}(a_k|s_k)}{{\pi }_{{\theta }_{\mathrm{old}}}(a_k|s_k)}}\end{array}$$

(22)

From equation (22), we can find that in the optimization of each episode, the algorithm first obtain the second derivative of the constraint and then optimize through conjugate gradient. The process is time-consuming, and then we utilize a method, which only requires the first derivative during the optimization. Set

$$\begin{array}{c}\mathrm{Tanh}(x)=\mu \times \frac{e^{\nu (x-1)}-e^{-\nu (x-1)}}{e^{\nu (x-1)}+e^{-\nu (x-1)}}+1\hfill \end{array}$$

(23)

where $\mu$ and $\nu$ are user-defined hyperparameters. Based on equation (22), in our algorithm, we aim to optimize policy according to such equation

$$\underset{\theta }{\mathrm{maximize}}\mathbb{E}\left[\min (\mathcal{F}\left(\theta \right)A_{{\theta }_{\mathrm{old}}},\mathrm{Tanh}\left(\mathcal{F}\left(\theta \right)\right)A_{{\theta }_{\mathrm{old}}})\right]$$

(24)

Compared with equation (22), equation (24) only have upper bound with the variation of $\mathcal{F}(\theta )$ either when $A_{{\theta }_{\mathrm{old}}}>0$ or $A_{{\theta }_{\mathrm{old}}}<0$ . It means that both algorithms can efficiently improve policy when they find better advantages. However, equation (24) can also utilize the “bad” information to optimize policy. Simultaneously, equation (24) does not use the second derivative, which means its algorithm optimizes policy more quickly. In this paper, we will use equation (24) to train and optimize our controllers and we take hyperparameters as $\mu =0.2$ , $\nu =5$ .

Remark 5

In the TRPO algorithm, the policy is optimized through conjugate gradient algorithm and before that TRPO will obtain the second derivative of the constraint, which is time-consuming. However, algorithm in this paper is based on the TRPO abandon using the constraint, which means the algorithm will optimize policy only with first-order derivative. In other words, without constraints, maximization of the objective will lead to an excessively large policy update. When $A_{{\theta }_{\mathrm{old}}}>0$ or $A_{{\theta }_{\mathrm{old}}}<0$ , TRPO and our algorithm can both reach an upper bound or a lower bound, which are determined by the user-defined hyperparameters. Both algorithms utilize the change in probability ratio when it would make the objective improve, but the algorithm in this paper also includes it when it makes the objective worse. Based on these conditions, our algorithm can accomplish the same target as TRPO and optimize method more efficiently.

Controller implement

In this section, we introduce the implement of the critic–actor-based controller and conclude the algorithm.

From Figure 2, we have that the RL-based controller utilizes reward, state signals to update and output voltages. Reward obtained by reward function module guarantees the controller satisfy multiple constraints. Back propagation neural networks (NNs) are employed to implement the controller. For the critic part, we use a three-layer network with definition as follows

$$V_{\varphi }\left(s\right)=W_1^T\phi (W_2^{T}s+b_2)+b_1$$

(25)

where $W_1$ and $W_2$ are weighting matrices. $b_1$ and $b_2$ are bias. We use the ReLU activation function $\phi (·)$ for NN. For elevation axis, we set the input state as $s_e(t)=[\stackrel{·}{\alpha }(t{)}^{\mathrm{T}}{e}_e(t{)}^{\mathrm{T}}{]}^{\mathrm{T}}$ . The output layer is the value function $V_{{\varphi }_e}(s_e(t))$ .

For the critic part, our aim is to minimize the advantage function $A_{{\theta }_{\mathrm{old}}}(s_t)$ . Here, we have

$$A_{{\theta }_{\mathrm{eold}}}\left(s_e\left(t\right)\right)=\sum _{t\prime >t}{\gamma }^{t\prime -t}{r}_e(t\prime )-V_{{\varphi }_e}\left(s_e\left(t\right)\right)$$

With minibatch $T$ , we define

$$J_{\mathrm{ecritic}}\left({\varphi }_e\right)=-\sum _{t=1}^T{(\sum _{t\prime >t}{\gamma }^{t\prime -t}{r}_e(t\prime )-V_{{\varphi }_e}\left(s_t\right))}^2$$

(26)

We can update the critic parameters ${\varphi }_e$ by the gradient method according to $J_{\mathrm{ecritic}}({\varphi }_e)$ . Similarly, for the pitch axis, the inputs are $s_p(t)=[\stackrel{·}{\beta }(t{)}^{\mathrm{T}}{e}_p(t{)}^{\mathrm{T}}{]}^{\mathrm{T}}$ and value function is $V_{{\varphi }_p}(s_p(t))$ . We update parameters ${\varphi }_p$ by optimizing $J_{\mathrm{pcritic}}({\varphi }_p)$ . In this part, the value function will describe the reward environment more and more accurate.

For the actor network, we optimize the actor network by using the advantage function value obtained from the critic part. In this paper, actor network structure is similar to critic network, but it has two outputs. The inputs for the network are $s_e(t)$ for elevation axis and $s_p(t)$ for pitch axis. The outputs for each axis consist of two parts: $u_{\mathrm{mean}}(t)$ and $u_d(t)$ . The control input $u(t)$ is sampled under a normal distribution with the mean value $u_{\mathrm{mean}}(t)$ and the standard deviation $u_d(t)$ . The control outputs for the elevation and pitch axis are voltage $V_s$ and $V_d$ . Thus, we have

$$\begin{array}{c}{u}_{\mathrm{mean}}=W_{1m}^T{\phi }_2(W_2^{T}s+b_2)+b_{1m}\hfill \\ \multicolumn{1}{c}{u_d={\phi }_1\left(W_{1d}^T{\phi }_2(W_2^{T}s+b_2)\right)+b_{1d}}\end{array}$$

(27)

where $W_{1m}$ , $W_{1d}$ and $W_2$ are weighting matrices, and $b_{1m}$ , $b_{1d}$ and $b_2$ are bias. ${\phi }_1(·)$ and ${\phi }_2(·)$ are the softplus and Rectified Linear Unit (ReLU) activation functions.

In this part, our aim is to optimize the actor policy as fast as possible according to equation (24). For elevation axis, we define

$$J_{\mathrm{actor}}\left(\theta \right)=\sum _{t=1}^T\min (\mathcal{F}\left(\theta \right)A_{{\theta }_{\mathrm{old}}}\left(t\right),\mathrm{Tanh}\left(\mathcal{F}\left(\theta \right)\right)A_{{\theta }_{\mathrm{old}}}\left(t\right))$$

(28)

We can obtain $A_{{\theta }_{\mathrm{old}}}(t)$ from the critic part, and then update the actor network parameters $\theta$ by the gradient method according to $J_{\mathrm{actor}}(\theta )$ . For elevation axis, the input is $s_e(t)$ and advantage function is $A_{{\theta }_{\mathrm{eold}}}(t)$ . Similarly, for the pitch axis, the input is $s_p(t)$ and advantage function is $A_{{\theta }_{\mathrm{pold}}}(t)$ obtained from the pitch critic network. We update parameters ${\theta }_e$ and ${\theta }_p$ by optimizing $J_{\mathrm{actor}}({\theta }_e)$ and $J_{\mathrm{actor}}({\theta }_p)$ .

The technique used in the article is a gradient descent method. It is well known that optimizer is important in dealing with the gradient descent problem. In this paper, we utilize a momentum optimizer, Adam optimizer, which was brought out by Kingma and Ba. ³⁷ It has lower variance than the Stochastic gradient descent (SGD) optimizer when the optimization converges.

Based on the abovementioned statement, we have our controller design algorithm described in the following steps:

Algorithm: RL controller design method
1. Set the hyperparameters in reward function module and equation (24).
2. Initialize the parameters of two NNs, learning rates and minibatch size. Run the system and training.
3. In the minibatch period $T$ , run the system with the policy ${\pi }_{{\theta }_{\mathrm{old}}}$ . Collect state vectors, system inputs and reward.
4. Calculate the advantage function value $A_{{\theta }_{\mathrm{old}}}(s)$ and optimize the critic network with the objective function $J_{\mathrm{critic}}(\varphi )$ in equation (26).
5. Optimize the actor network with the objective function $J_{\mathrm{actor}}(\theta )$ in equation (28) and get the new policy ${\pi }_{{\theta }_{\mathrm{new}}}$ .
6. Justify the sum reward of this episode is qualified or not. If it is qualified, stop training and obtain the parameters of the networks. If the sum reward is large, go back to the Step 3 with running the next $T$ with policy ${\pi }_{{\theta }_{\mathrm{new}}}$ .

Example

In this section, we will use the experiments on the table-mount helicopter platform to illustrate the applicability of the proposed algorithm.

The equipment is shown in Figure 1, and the control system model is shown in Figure 2. The sampling time for the experiments is 5 ms. The parameters of the helicopter are shown in Table 2. The hidden layers of both critic network and actor network have 20 neurons. The minibatch for the training technique is 64. The learning rates for critic net and actor net are 0.0001 and 0.0002, respectively. The discounted rate $\gamma$ is 0.95. In table-mount helicopter platform, we construct the controller in the Simulink and then the software compiles and downloads codes into the practical system. The first step here is to establish a numerical simulation environment by using the parameters in Table 2. Then, the proposed algorithm is utilized to train the controllers. After controllers satisfy the strict limitation constraints, we load the net parameters to the practical platform and then controllers will be trained in the practical environment.

Table 2.

Parameters of the Quanser helicopter.

Parameter	Value	Parameter	Value
$J_e$	1.034 kg m2	$J_p$	0.045 kg m2
$l_{\alpha }$	0.66 m	$l_{\beta }$	0.178 m
$m$	0.094 kg	$g$	9.81 m/s2
$K_f$	0.1188 N/V

Experiment with constrained control input

In this experiment, we test the applicability of our controllers with strict limitation constraint. In the table-mount helicopter plant, the initial position of the helicopter is −27° elevation axis and initial pitch angle is 0°. The targets of elevation axis and pitch axis are both 0°. In the traditional method, we cannot control the maximum of the control input and we adjust the parameters of the controller in practice. In this experiment, we provide a LQR controller with little parameter adjustments for comparison. The controller parameters are shown in equation (29). Assume there are strong limitations for DC motor such that voltages $V_f$ and $V_b$ should be in the range $[1,24]$

$$K_e=\left[\begin{array}{cc}-37.67\hfill & -20.95\hfill \end{array}\right],K_p=\left[\begin{array}{cc}13.21\hfill & -4.769\hfill \end{array}\right]$$

(29)

According to the algorithm, we first establish the reward function module with $e_m=0.45$ for elevation axis and $e_m=0.2$ for pitch axis

$$\begin{array}{cc}\multicolumn{1}{c}{r_{\mathrm{ea}}\left(k\right)}& =-|e_e\left(k\right)|,r_{\mathrm{pa}}\left(k\right)=-|e_p\left(k\right)|\hfill \\ \multicolumn{1}{c}{w_{e1}=}& \{\begin{array}{cc}1\hfill & \mathrm{if}{r}_{\mathrm{ac}1}\left(k\right)<0\hfill \\ \multicolumn{1}{c}{0.1}& \mathrm{otherwise}\hfill \end{array},w_{p1}=\{\begin{array}{cc}1\hfill & \mathrm{if}{r}_{\mathrm{pc}1}\left(k\right)<0\hfill \\ \multicolumn{1}{c}{0.1}& \mathrm{otherwise}\hfill \end{array}\hfill \end{array}$$

$$\begin{array}{cc}\multicolumn{1}{c}{r_{\mathrm{ec}1}\left(k\right)}& =\{\begin{array}{cc}-0.4-0.093\times (V_s\left(k\right)-48)\hfill & \mathrm{if}{V}_{\mathit{s}}\left(k\right)>48\hfill \\ \multicolumn{1}{c}{-0.3-0.093\times (5-V_s\left(k\right))}& \mathrm{if}{V}_{\mathit{s}}\left(k\right)<5\hfill \\ \multicolumn{1}{c}{0}& \mathrm{otherwise}\hfill \end{array}\hfill \\ \multicolumn{1}{c}{r_{\mathrm{pc}1}\left(k\right)}& =\{\begin{array}{cc}-0.2-0.2\times (V_d\left(k\right)-5)\hfill & \mathrm{if}{V}_{\mathit{d}}\left(k\right)>5\hfill \\ \multicolumn{1}{c}{-0.2-0.2\times (-5-V_d\left(k\right))}& \mathrm{if}{V}_{\mathit{d}}\left(k\right)<-5\hfill \\ \multicolumn{1}{c}{0}& \mathrm{otherwise}\hfill \end{array}\hfill \end{array}$$

(30)

For elevation axis, the reward at state $s_e(k)$ is $r_e(k)=r_{\mathrm{ea}}(k)+w_{e1}{r}_{\mathrm{ec}1}(k)$ and for pitch axis $r_p(k)=r_{\mathrm{pa}}(k)+w_{p1}{r}_{\mathrm{pc}1}(k)$ . Compared with the LQR controllers, we can obtain Figures 3 and 4.

Figure 3.

Comparison of elevation axis tracking response for (a) reinforcement learning-based controllers and (b) traditional LQR controller with small parameter adjustment.

Figure 4.

Comparison of control output $V_s$ for (a) reinforcement learning-based controllers and (b) traditional LQR controller with small parameter adjustment.

From Figure 3, compared with the LQR controller, the controller design in our RL-based algorithm has a shorter settling time, less overshoot and less steady-state error. At the same time, our controllers are model-free, which means they can learn to optimize themselves without any parameter adjustments. Also, from Figure 4(a), we can find that under this reward environment, controllers learn to adapt this penalty environment, which satisfies limitation.

Experiment with multiple constraints

During this experiment, we will illustrate that the proposed controllers can guarantee the special performance need. Similar to the abovementioned experiment, the initial positions for elevation axis and pitch axis are −27° and 0°. The targets of two axes are both 0°. In this experiment, we want the helicopter to have such a performance that there is no overshoot in the step response. We add the special reward as follows with $t_{\exp }=2$ , $e_m=0.45$ for elevation axis and $e_m=0.2$ for pitch axis

$$\begin{array}{cc}{w}_{e2}=\hfill & \{\begin{array}{cc}1\hfill & \mathrm{if}{r}_{\mathrm{ac}1}\left(k\right)<0\hfill \\ \multicolumn{1}{c}{0.1}& \mathrm{otherwise}\hfill \end{array},w_{p2}=\{\begin{array}{cc}1\hfill & \mathrm{if}{r}_{\mathrm{pc}1}\left(k\right)<0\hfill \\ \multicolumn{1}{c}{0.1}& \mathrm{otherwise}\hfill \end{array}\hfill \\ \multicolumn{1}{c}{r_{\mathrm{ec}2}\left(k\right)}& =\{\begin{array}{cc}100\times \stackrel{·}{\alpha }\left(k\right)\hfill & (\mathrm{if}\stackrel{·}{\alpha }\left(k\right)>0.225)\hfill \\ \multicolumn{1}{c}{0}& \mathrm{otherwise}\hfill \end{array}\hfill \\ \multicolumn{1}{c}{r_{\mathrm{pc}2}\left(k\right)}& =\{\begin{array}{cc}100\times \stackrel{·}{\beta }\left(k\right)\hfill & (\mathrm{if}\stackrel{·}{\beta }\left(k\right)>0.1)\hfill \\ \multicolumn{1}{c}{0}& \mathrm{otherwise}\hfill \end{array}\hfill \end{array}$$

(31)

According to equations (30) and (31), we can have the reward for elevation axis as $r_e(k)=r_{\mathrm{ea}}(k)+w_{e1}{r}_{\mathrm{ec}1}(k)+w_{e2}{r}_{\mathrm{ec}2}(k)$ and for pitch axis $r_p(k)=r_{\mathrm{pa}}(k)+w_{p1}{r}_{\mathrm{pc}1}(k)+w_{p2}{r}_{\mathrm{pc}2}(k)$ . The main aim of this part is to test whether proposed controllers can guarantee the performance which has no overshoot. We only observe the elevation axis and obtain Figures 5 and 6.

Figure 5.

Front motor and back motor voltages of the step response for elevation axis.

Figure 6.

Step response for the elevation axis: (a) before training, (b) during the training and (c) after training.

Figure 6 shows the tracking response for the elevation axis before, during and after training. The controller before training we used is the controller in the experiment 1, which is trained in the reward environment (30). The settling time is around 2 s, the overshot is 4.5% and the tracking error is about 0.34°. The tracking response (Figure 6(b)) we got utilizes the controller, which we trained in the reward environments (30) and (31) after 500 iterations. The settling time is 2.9 s, and it has no overshot but −0.8° tracking error. After the complete training, we obtain the tracking response in Figure 6(c), whose settling time is 2.5 s and tracking error is 0.16°. Surely, the response has no overshoot.

Figure 5 shows the front motor voltage and back motor voltage of the tracking response after training. Both voltages are in the interval (0, 24), which satisfies the reward constraints $r_{\mathrm{ec}1}$ and $r_{\mathrm{pc}1}$ .

Experiment with step signal tracking

In this part, we illustrate that our two-axis controllers can successfully track the step signals. The initial position of the helicopter is −27° for elevation axis and 0° for pitch axis. We will carry out four experiments, and the target pitch angle we set remains 20°. For elevation axis, the helicopter will carry out four experiments separately, and the four target elevation angles are −22°, −17°, −12° and −7°. The reward function module we utilized in this part is that $r_e(k)=r_{\mathrm{ea}}(k)+w_{e1}{r}_{\mathrm{ec}1}(k)+w_{e2}{r}_{\mathrm{ec}2}(k)$ for elevation axis and $r_p(k)=r_{\mathrm{pa}}(k)+w_{p1}{r}_{\mathrm{pc}1}(k)+w_{p2}{r}_{\mathrm{pc}2}(k)$ for pitch axis. Through training for the table-mount helicopter, results are shown in Figures 7 and 8.

Figure 7.

(a) Elevation axis tracking responses for four cases and (b) elevation axis control output $V_s$ voltage for four cases.

Figure 8.

(a) Tracking responses for pitch axis from 0° to 20° and (b) control output $V_d$ voltage.

As shown in Figure 7(a), the controllers we design can successfully help the helicopter track the signals in different step cases. Figure 7(b) shows the control output $V_s$ voltage, which determines the elevation axis motion. Similarly, Figure 8(a) shows the tracking response for the pitch axis and Figure 7(b) the control output $V_d$ voltage. One can see that Figures 7 and 8 illustrate that the proposed controllers can successfully track the set-point signals and satisfy our special constraints.

Conclusion

In this paper, we propose an RL-based controller to address the controller problem for a 3-DOF helicopter with multiple constraints. By choosing a special reward function module, the critic controller can successfully learn the reward environment and actor controller can update policy quickly. There are two advantages of this algorithm. One is that the controller we design is model-free controller, which means the controller is trained only according to the control inputs and system states. This point results in that the controllers will update by themselves by using real-time data. Another advantage is that we can design controllers according to our demand through reward function module. The platform experiment results show that the proposed controllers can successfully satisfy the constraints, special performance and step signal tracking performance.