Adaptive optimal control approach to robust tracking of uncertain linear systems based on policy iteration

Abstract

In this study, an optimal adaptive control approach is established to solve the robust output tracking problem of a class of continuous time uncertain linear systems based on the policy iteration (PI) in actor-critic algorithm. First, by augmenting the integral variables of the tracking error into state variables, the robust tracking problem is transformed into a robust control problem of an augmented uncertain linear system. It is proven that the robust control law of the augmented system enables the output of the considered system to track a polynomial time signal asymptotically. Second, an optimal control method in the corresponding auxiliary nominal system is established, and based on the Bellman optimality principle, PI algorithms are proposed to solve online tracking controllers for the matched and the mismatched uncertain systems. Finally, for testing the availability of the proposed approach and theoretical results, two numerical experiments are provided.

Introduction

Control systems in many practical problems have uncertainties, owing to data errors or disturbances. ¹ Therefore, study on the robust control problem of uncertain systems has garnered significant attention. An adaptive optimal control method to design a robust tracking controller based on the policy iteration (PI) of the actor-critic algorithm is introduced in the research of algorithm design. Adaptive optimal control is a control method that automatically adjusts controller’s parameters to optimize the performance of the control system according to the change of parameters or environment. Moreover, it is widely used in control practice. ²

Since the 1980s, the robust control problem of practical uncertain systems has been a research hotspot. Algebraic Riccati equations (ARE) were utilized to propose a robust control design for some uncertain systems.^3,4 The robust control of time-varying uncertain linear systems was studied, ⁵ and the controller parameters were obtained using a one-dimensional search method. The proposed method could be effective if the system were to have satisfied the matching condition. Using a noniterative method, a robust controller was obtained for a matched system. ⁶ The need for pre-compensators of the unstable nominal system was a disadvantage of this design method. In addition, other effective robust control-design methods were proposed for general matched uncertain systems.^7,8 However, only a few study results are available for mismatched systems having an uncertainty in the input matrix.

The robust tracking problem has been investigated in many studies.^9–12,32,34 Schmitendorf et al. proposed a robust tracking control approach for time-invariant uncertain system with unknown constant disturbances. ⁹ Benson and Schmitendorf considered a robust tracking problem by using observer-based augmented system. ¹⁰ Utilizing an improved linear quadratic optimal control method, Shieh et al. designed a robust tracking law for an uncertain linear systems. ¹¹ Solving a robust tracking problem for an uncertain system was turned into an optimal control problem. ¹² For linear systems, it is a linear regulator problem, which is solved using an ARE.

Furthermore, the PI in reinforcement learning (RL) has been utilized extensively in solving control and tracking problems for deterministic systems. In terms of feedback control, we can refer to the literature.^13–15 There are also some literatures on using PI algorithm to solve tracking problem. An online RL was established to obtain a linear tracking controller in partially unknown linear systems with unstable tracking signals. ¹⁶ By constructing an augmented system comprising the state variable and tracking signal, the tracking problem was turned into an optimal control problem. Therefore, the optimal tracking cost is quadratic as well. An adaptive dynamic programming (ADP) algorithm was proposed to obtain the tracking controller for completely unknown linear systems, ¹⁷ where tracking signals were generated for stabilizing the systems. Moreover, RL and ADP techniques were employed to achieve an optimal output regulation for linear systems. ¹⁸

On the other hand, the application of RL in the robust control of uncertain systems has been proposed by many researchers.^19–26 Robust tracking problems exist widely in practical applications. However, to our knowledge, only a few studies were conducted on the PI-based robust tracking of uncertain linear systems. Most existing design methods for solving robust tracking problems are based on impractical nominal systems. Therefore, it is necessary to use the PI algorithm to solve the robust tracking problems of uncertain linear systems with an unknown nominal system matrix. In this study, a PI method was developed to solve the robust tracking control of an uncertain linear system with a polynomial tracking signal. The robust tracking problem was turned into a robust control problem of an augmented linear system. Additionally, online PI algorithms were developed to solve the considered tracking design problem for matched and mismatched systems.

Our main contributions are as follows. First, we considered the tracking problem of general uncertain linear systems in which both the state matrix and the input matrix were uncertain. In the existing literature, ¹² the design of a robust tracking-control method was proposed for uncertain linear systems only when an uncertainty entered the system matrix. The existing results were extended to a case where the input matrix was uncertain. Second, online PI algorithms were presented to solve tracking problems for the matched uncertain system and the mismatched uncertain systems. Because it is practically difficult to obtain the nominal system-matrix information accurately, using the PI algorithm is advantageous. As a result, we extended the PI algorithm to calculate robust tracking control law for the general uncertain linear systems.

The rest of this paper is arranged as follows. We formulate the robust tracking problems and propose some basic results for the issues under consideration in Section 2. Solving robust tracking problem is converted to calculate a robust control law for augmented uncertain systems. In Sections 3 and 4, the robust tracking problem for a matched and mismatched linear system is solved by transforming it into a robust control problem of an augmented system. Online PI algorithms are developed for an augmented uncertain system based on the optimal control of an auxiliary linear system. To support the proposed theoretical framework, we provide numerical experiments with two examples in Section 5. In Section 6, the study is concluded, and the scope for future research is discussed.

Robust tracking control framework

Consider an continuous-time linear system with uncertainty as follows:

$$\begin{array}{c}\stackrel{·}{x}=A(s)x+B(l)u\hfill \\ y=\mathrm{Cx},\hfill \end{array}$$

(1)

where $x\in R^n$ is the state vector, $u\in R^m$ is the input variable, $y\in R^1$ is the system output, $s\in \mathbb{S},l\in \mathbb{L}$ are the uncertain parameter vectors, and $\mathbb{S}$ and $\mathbb{L}$ are sets of uncertain parameters. $A(s)$ is $n\times n$ uncertain state parameter matrix, $B(l)$ is $n\times m$ uncertain input parameter matrix, and C is an $1\times n$ constant output matrix.

The objective of control design is to establish a control input, $u=\mathrm{Kx}$ , such that the system output y asymptotically tracks the desired referenced signal, $y_r$ , for all $s\in \mathbb{S}$ and $l\in \mathbb{L}$ . The referenced signal is assumed to be a polynomial time signal, $y_r=a_0+a_{1}t+\cdots +a_{d-1}{t}^{d-1}$ , where $a_0,a_1,\cdots ,a_{d-1}$ are constants, and d is a nonnegative integer. In particular, the control design goal is to establish a control input, $u=\mathrm{Kx}$ , such that the closed-loop uncertain system (1) is asymptotically stable and the output, $y=\mathrm{Cx}$ , can asymptotically track the signal, $y_r$ , for all $s\in \mathbb{S}$ and $l\in \mathbb{L}$ .

Some definitions and basic assumptions^1,27 are elaborated as follows.

• Assumption 1. Nominal values $s_0\in \mathbb{S}$ and $l_0\in \mathbb{L}$ , such that $(A(s_0),B(l_0))$ is stabilizable and $(A(s_0),C)$ observable.

• Definition 1. System (1) satisfies matched condition in system matrix if, for every $s\in \mathbb{S}$ , there is a matrix, $\phi (s)$ , such that

$$A(s)-A(s_0)=B(l_0)\phi (s),$$

(2)

where $\phi (s)\in R^{m\times n}$ .

• Definition 2. System (1) satisfies the matched condition in input matrix if, for every $l\in \mathbb{L}$ , there is a matrix $\overline{\phi }(l)$ , such that

$$B(l)-B(l_0)=B(l_0)\overline{\phi }(l),$$

(3)

where $\overline{\phi }(l)\in R^{m\times m}$ , and $\overline{\phi }(l)\ge 0$ .

• Definition 3. When system (1) satisfies conditions (2) and (3), for all $s\in \mathbb{S}$ and $l\in \mathbb{L}$ , it is called a matched uncertain linear system.

• Assumption 2. For the linear system (1) with matching uncertainties satisfying conditions (2) and (3), there is a known matrix, M, such that ${\phi }^T(s)\phi (s)\le M\ge 0$ for every $s\in \mathbb{S}$ .

If system (1) does not satisfy matching conditions (2) and (3), the pseudo inverse, $B(l_0{)}^{+}$ , of nominal input matrix $B(l_0)$ is introduced to decompose the uncertain system matrix and input the matrix into the following matched and mismatched parts.

$$\begin{array}{c}A(s)-A(s_0)=B(l_0)B(l_0{)}^{+}[A(s)-A(s_0)]\hfill \\ +[I-B(l_0)B{(l_0)}^{+}][A(s)-A(s_0)],\hfill \end{array}$$

(4)

and

$$\begin{array}{c}B(l)-B(l_0)=B(l_0)B(l_0{)}^{+}[B(l)-B(l_0)]\hfill \\ +[I-B(l_0)B{(l_0)}^{+}][B(l)-B(l_0)],\hfill \end{array}$$

(5)

where $B(l_0{)}^{+}={[B^T(l_0)B(l_0)]}^{-1}{B}^T(l_0)$ .

If system (1) is a mismatched uncertain system, let the uncertainty of the system be subject to the following assumptions.

• Assumption 3. A positive semidefinite matrix, F, exists, such that

$${[A(s)-A(s_0)]}^T[B(l_0{)}^{+}{]}^{T}B(l_0{)}^{+}[A(s)-A(s_0)]\le F.$$

(6)

• Assumption 4. A positive semidefinite matrix, H, exists, such that

$${[A(s)-A(s_0)]}^T[A(s)-A(s_0)]\le H.$$

(7)

• Assumption 5. A positive semidefinite matrix, G, exists, such that

$${[A(s)-A(s_0)]}^T{C}^{T}C[A(s)-A(s_0)]\le G.$$

The above assumptions are common in the study of uncertain systems^1,26,27.

To solve the robust tracking problem, we introduce a new variable, $e=y-y_r$ , as an error. Based on the integrals of the error variable, the following new state variables are defined:

$${\stackrel{·}{q}}_1=e,{\stackrel{·}{q}}_2=q_1,\cdots ,{\stackrel{·}{q}}_d=q_{d-1}$$

Now, define the augmented system state as

$$X(t)={\left[\begin{array}{ccccc}{x}^T(t)\hfill & q_1\hfill & q_2\hfill & \cdots \hfill & q_d\hfill \end{array}\right]}^T$$

Using the augmented system state, an uncertain augmented linear system is constructed as

$$\stackrel{·}{X}=T(s)X+\overline{B}(l)u+N{y}_r,$$

(8)

where

$$\begin{array}{cc}T(s)\hfill & =\left[\begin{array}{cccccc}A(s)\hfill & \hfill & \hfill & O_{n\times d}\hfill & \hfill & \hfill \\ \multicolumn{1}{c}{C}& \hfill & \hfill & O_{1\times d}\hfill & \hfill & \hfill \\ \multicolumn{1}{c}{0}& 1\hfill & 0\hfill & \dots \hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{⋮}& ⋮\hfill & ⋮\hfill & \hfill & ⋮\hfill & ⋮\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill & \dots \hfill & 1\hfill & 0\hfill \end{array}\right],\hfill \\ \multicolumn{1}{c}{\overline{B}(l)}& =\left[\begin{array}{c}B(l)\hfill \\ \multicolumn{1}{c}{O_{d\times m}}\end{array}\right],N=\left[\begin{array}{c}{O}_{n\times 1}\hfill \\ \multicolumn{1}{c}{-1}\\ \multicolumn{1}{c}{0}\\ \multicolumn{1}{c}{⋮}\\ \multicolumn{1}{c}{0}\end{array}\right].\hfill \end{array}$$

Here, $O_{i\times j}\in R^{i\times j}$ represents a zero matrix. Equation (8) can be regarded as a linear nonhomogeneous uncertain system with a nonhomogeneous term, $N{y}_r$ . The considered robust tracking problem could be converted to the following robust stabilization problem. For every $s\in \mathbb{S}$ and $l\in \mathbb{L}$ , find a control input, $u=\mathrm{KX}$ , such that the augmented homogeneous uncertain linear system

$$\stackrel{·}{X}=T(s)X+\overline{B}(l)u$$

(9)

is asymptotically stable.

• Lemma 1. If system (1) satisfies conditions (2) and (3), then the augmented uncertain system (9) is a matched uncertain linear system.

Proof. Assume that system (1) satisfies the system matrix matched condition (2) and the input matrix matched condition (3). Consequently, $A(s)-A(s_0)=B(l_0)\phi (s)$ . Therefore,

$$\begin{array}{c}T(s)-T(s_0)=\left[\begin{array}{cc}A(s)-A(s_0)\hfill & O_{n\times d}\hfill \\ \multicolumn{1}{c}{O_{d\times n}}& O_{d\times d}\hfill \end{array}\right]\hfill \\ =\left[\begin{array}{c}B(l_0)\hfill \\ \multicolumn{1}{c}{O_{d\times m}}\end{array}\right]\left[\begin{array}{cc}\phi (s)\hfill & O_{m\times d}\hfill \end{array}\right]\hfill \\ =\overline{B}(l_0)\mathrm{\Phi }(s),\hfill \end{array}$$

where $\overline{B}(l_0)=\left[\begin{array}{c}B(l_0)\hfill \\ \multicolumn{1}{c}{O_{d\times m}}\end{array}\right]\in R^{(n+d)\times m}$ , and $\mathrm{\Phi }(s)=\left[\begin{array}{c}\phi (s)O_{m\times d}\hfill \end{array}\right]\in R^{m\times (n+d)}$ . This implies that the augmented system (3) satisfies the system matrix-matched condition. Moreover,

$$\begin{array}{c}\overline{B}(l)-\overline{B}(l_0)=\left[\begin{array}{c}B(l)-B(l_0)\hfill \\ \multicolumn{1}{c}{O_{d\times m}}\end{array}\right]\hfill \\ =\left[\begin{array}{c}B(l_0)\overline{\phi }(l)\hfill \\ \multicolumn{1}{c}{O_{d\times m}}\end{array}\right]=\overline{B}(l_0)\overline{\phi }(l),\hfill \end{array}$$

which implies that the augmented system (9) satisfies the input matrix-matched condition. Hence, the proof is complete.

• Lemma 2. Denote the new state as $q={\left[\begin{array}{cccc}{q}_1\hfill & q_2\hfill & \cdots \hfill & q_d\hfill \end{array}\right]}^T$ . For all $s\in \mathbb{S},l\in \mathbb{L}$ , supposing that the state feedback controller $u=\mathrm{KX}=\left[\begin{array}{cc}\overline{K}x\hfill & K_{1}q\hfill \end{array}\right]$ can stabilize the system $\stackrel{·}{X}=T(s)X+\overline{B}(l)u$ , then $u=\overline{K}x$ will stabilize system (1), and $y\to y_r$ as $t\to \infty$ .

Proof. Because (9) is an augmented system corresponding to system (1), $u=\overline{K}x$ will stabilize system (1). By determining the d-order differential on both sides in the augmented system (8), it yields

$$\begin{array}{c}{X}^{(d+1)}=T(s)X^{(d)}+\overline{B}(l)K{X}^{(d)}\hfill \\ =(T(s)+\overline{B}(l)K)X^{(d)}.\hfill \end{array}$$

(10)

Considering that $u=\mathrm{KX}$ can stabilize system (9), the matrix $T(s)+\overline{B}(l)K$ is Hurwitz stable. It follows from (10) that $X^{(d)}\to 0$ as $t\to \infty$ . This implies that $q_d^{(d)}=e\to 0$ as $t\to \infty$ . Therefore, we have $y\to y_r$ as $t\to \infty$ . Hence, the proof is complete.

Lemma 2 shows that the robust tracking problem of system (1) can be transformed to a robust stabilization problem of the augmented homogeneous system (9).

• Remark 1. Tan et al. ¹² considered the robust tracking of an uncertain linear system without input uncertainty. In this study, we considered a system with uncertainties entering the input matrix, which is an extension of the existing study outcomes.

Here, the system is divided into matched and mismatched cases, and the robust tracking problem of the system is discussed.

Matched uncertain linear systems

In this section, PI algorithms are developed to calculate robust tracking control law for linear systems with matched uncertainties. The problem is transferred into stabilizing an augmented uncertain system which contains the original system states and tracking signal. Based on solving an optimal control problem with augmented nominal system and predefined performance index, the PI algorithms are proposed to obtain robust tracking feedback control.

Robust Stabilization of an augmented linear system with uncertainty

Here, we discuss system (1) when it satisfies the matched conditions. Furthermore, robust stabilization for the augmented uncertain systems is transformed into calculating an optimal control with nominal system and predefined performance index. The optimal control problem is solved using the PI method, and the required robust tracking control law is obtained.

For the nominal linear system

$$\stackrel{·}{X}=T(s_0)X+\overline{B}(l_0)u,$$

(11)

we construct an optimal-control problem. Acquire a controller, $u=\mathrm{KX}$ , that minimizes the following performance index

$$V(X(t_0),u(.))={\int }_{t_0}^{\infty }[X^T\mathrm{QX}+u^{T}u]\mathrm{dt},$$

(12)

where $Q=\overline{M}+I\ge 0$ and $\overline{M}$ is the supremum of uncertainty ${\mathrm{\Phi }}^T(s)\mathrm{\Phi }(s)$ , and I is an identity matrix with proper dimensions. Actually, $\overline{M}=\left[\begin{array}{cc}M\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]$ , which based on the following derivation

$$\begin{array}{c}{\mathrm{\Phi }}^T(s)\mathrm{\Phi }(s)={\left[\begin{array}{cc}\phi (s)\hfill & O_{m\times d}\hfill \end{array}\right]}^T\left[\begin{array}{cc}\phi (s)\hfill & O_{m\times d}\hfill \end{array}\right]\hfill \\ =\left[\begin{array}{cc}{\phi }^T(s)\phi (s)\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]\hfill \\ \le \left[\begin{array}{cc}M\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]\hfill \\ =\overline{M}\hfill \end{array}$$

Here, O denotes a zero matrix of an appropriate dimensions.

According to the optimal control theory, ²⁶ $u=-{\overline{B}}^T(l_0)\mathrm{PX}$ is the solution of the optimal control problem (11) and (12), which the positive matrix P satisfies the following ARE:

$$T^T(s_0)P+\mathrm{PT}(s_0)+Q-P\overline{B}(l_0){\overline{B}}^T(l_0)P=0.$$

(13)

• Theorem 1. Denote $K=-{\overline{B}}^T(l_0)P$ . If system (1) is a matched uncertain system with conditions (2) and (3), then the solution $u=\mathrm{KX}$ , in optimal control problem (11) and (12) can stabilize the augmented uncertain system (9). That is, for every $s\in \mathbb{S},l\in \mathbb{L}$ , the closed-loop system,

$$\stackrel{·}{X}=[T(s)+\overline{B}(l)K]X$$

(14)

is asymptotically stable.

Proof. It follows from Lemma 1 that the augmented system (9) satisfies the following matching conditions:

$$T(s)-T(s_0)=\overline{B}(l_0)\mathrm{\Phi }(s)$$

(15)

and

$$\overline{B}(l)-\overline{B}(l_0)=\overline{B}(l_0)\overline{\phi }(l)$$

(16)

Choosing the Lyapunov function as $V(X)=X^T\mathrm{PX}$ and taking its time derivative, along the closed-loop system (14), we obtain

$$\begin{array}{c}\frac{\mathrm{dV}}{\mathrm{dt}}={\stackrel{·}{X}}^T\mathrm{PX}+X^{T}P\stackrel{·}{X}\hfill \\ =X^T[T(s)+\overline{B}(l)K{]}^T\mathrm{PX}+X^{T}P[T(s)+\overline{B}(l)K]X.\hfill \end{array}$$

By matched conditions (15) and (16),

$$\begin{array}{c}\frac{\mathrm{dV}}{\mathrm{dt}}=X^T[(T^T(s_0)P+\mathrm{PT}(s_0)+{\mathrm{\Phi }}^T(s){\overline{B}}^T(l_0)P\hfill \\ +K^T{\overline{B}}^T(l_0)P+K^T{\overline{\phi }}^T(l){\overline{B}}^T(l_0)P\hfill \\ +P\overline{B}(l_0)\mathrm{\Phi }(s)+P\overline{B}(l_0)K+P\overline{B}(l_0)\overline{\phi }(l)K]X.\hfill \end{array}$$

.

It follows from the optimal control gain, $K=-{\overline{B}}^T(l_0)P$ , and Riccati equation (13), that

$$\begin{array}{c}\frac{\mathrm{dV}}{\mathrm{dt}}=-X^T\overline{M}X-X^{T}X-X^T{K}^T\mathrm{KX}\hfill \\ -2X^T{K}^T\mathrm{\Phi }(s)X-2X^T{K}^T\overline{\phi }(l)\mathrm{KX}\hfill \\ =-X^T\overline{M}X-X^{T}X-X^T{K}^T\mathrm{KX}\hfill \\ -2X^T{K}^T\mathrm{\Phi }(s)X-2X^T{K}^T\overline{\phi }(l)\mathrm{KX}\hfill \\ -X^T{\mathrm{\Phi }}^T(s)\mathrm{\Phi }(s)X+X^T{\mathrm{\Phi }}^T(s)\mathrm{\Phi }(s)X\hfill \\ =-X^T(\overline{M}-{\mathrm{\Phi }}^T(s)\mathrm{\Phi }(s))X-2X^T{K}^T\overline{\phi }(l)\mathrm{KX}\hfill \\ -X^T(K+\mathrm{\Phi }(s){)}^T(K+\mathrm{\Phi }(s))X-X^{T}X.\hfill \end{array}$$

We immediately conclude that $\frac{\mathrm{dV}}{\mathrm{dt}}<0$ , because ${\mathrm{\Phi }}^T(s)\mathrm{\Phi }(s)\le \overline{M}$ and $\overline{\phi }(l)\ge 0$ . Therefore, for all $s\in \mathbb{S},l\in \mathbb{L}$ , closed-loop system (14) is asymptotically stable. Hence, the proof is complete.

PI algorithm and its convergence

This subsection has two parts. First, based on the performance function (12), an offline policy iteration algorithm is proposed to solve the robust tracking problem. Then, an integral reinforcement formula is developed based on the Bellman equation. An online real-time PI algorithm is developed to solve the robust tracking problem with unknown nominal system matrix.

For any initial time, t, rewrite the performance function (12) as

$$X^T\mathrm{PX}={\int }_t^{\infty }[X^T\mathrm{QX}+u^{T}u]\mathrm{dt}$$

(17)

Differentiating (17) with the trajectories of system (11) yields

$$H^{T}P+\mathrm{PH}+Q+K^{T}K=0,$$

(18)

where $H=T(s_0)+\overline{B}(l_0)K$ .

Using (18) and $K=-{\overline{B}}^T(l_0)P$ , an off-line PI algorithm for robust tracking problem is obtained.

• Algorithm 1. Offline PI algorithm for robust tracking of matched uncertain linear systems

• Policy evaluation: Solve $P_i$ in the equation, for a given control gain $K_i$ ,

  $$H_i^T{P}_i+P_i{H}_i+Q+K_i^T{K}_i=0,$$

  (19)

  where $H_i=T(s_0)+\overline{B}(l_0)K_i$ .

• Policy improvement: Compute $K_{i+1}$ using

$$K_{i+1}=-{\overline{B}}^T(l_0)P_i.$$

(20)

By alternative iterating (19) and (20), Algorithm 1 can be used to calculate the robust tracking law of uncertain systems, which is an extension of Kleinman’s algorithm. ²⁸ The convergence proof of Algorithm 1 is identical to that of Kleinman’s algorithm.

In Algorithm 1, we can solve ARE (13) by iteratively computing (19) and (20). However, the implementation of the algorithm needs to know the information of the nominal system, and the process of calculating the controller can only be realized offline. Here, an online PI algorithm is developed to solve ARE (13) with an unknown nominal system matrix, $T(s_0)$ .

For any initial time, t, the optimal cost in (12) can be rewritten as

$$\begin{array}{c}V(X(t))={\int }_t^{\infty }[X^T\mathrm{QX}+u^{T}u]\mathrm{dt}\hfill \\ ={\int }_t^{t+\mathrm{\Delta }t}[X^T\mathrm{QX}+u^{T}u]\mathrm{dt}\hfill \\ +{\int }_{t+\mathrm{\Delta }t}^{\infty }[X^T\mathrm{QX}+u^{T}u]\mathrm{dt},\hfill \end{array}$$

that is,

$$V(X(t))={\int }_t^{t+\mathrm{\Delta }t}[X^T\mathrm{QX}+u^{T}u]\mathrm{dt}+V(X(t+\mathrm{\Delta }t))$$

So we can have

$$\begin{array}{c}{X}^T(t)\mathrm{PX}(t)={\int }_t^{t+\mathrm{\Delta }t}[X^T\mathrm{QX}+u^{T}u]\mathrm{dt}\hfill \\ +X^T(t+\mathrm{\Delta }t)\mathrm{PX}(t+\mathrm{\Delta }t).\hfill \end{array}$$

(21)

In equation (21), only the matrix P is unknown. we can calculate the elements of the matrix P through the system trajectory data. Consequently, an online PI algorithm sloving robust tracking problem for uncertain linear system is obtained.

• Algorithm 2. The online PI algorithm for robust tracking of matched uncertain linear systems

• Policy evaluation: Compute $P_i$ from

  $$\begin{array}{c}{X}^T(t)P_{i}X(t)={\int }_t^{t+\mathrm{\Delta }t}[X^T\mathrm{QX}+X^T{K}_i^T{K}_{i}X]\mathrm{dt}\hfill \\ +X^T(t+\mathrm{\Delta }t)P_{i}X(t+\mathrm{\Delta }t).\hfill \end{array}$$

  (22)

• Policy improvement: Compute $K_{i+1}$

$$K_{i+1}=-{\overline{B}}^T(l_0)P_i.$$

(23)

• Remark 2. Algorithm 2 is an online PI algorithm based on the RL. Through the online trajectory data of system (11), the matrix $P_i$ can be solved using (22) and the least-squares method. Through iterative calculation and using (22) and (23), the robust control gain, K, for augmented uncertain linear system (9) is obtained. Moreover, according to Lemma 2, the system robust tracking law is expressed as $u=\overline{K}x$ by decomposing the robust control law into $\mathrm{KX}=\left[\begin{array}{cc}\overline{K}x\hfill & K_{1}q\hfill \end{array}\right]$ . As an advantage, Algorithm 2 need not know the nominal system matrix and can effectively avoid dimension disaster. In, ¹³ by the use of online PI, the linear quadratic regulator with unknown system matrix is calculated. We developed this algorithm to solve the robust tracking problem for an uncertain linear system.

The convergence in Algorithm 2 is proven as follows:

• Theorem 2. We assume that $(T(s_0)+\overline{B}(l_0)K_i)$ is stable, and we solve $P_i$ using equation (22), which equals to obtaining a solution for the following equation:

$$\begin{array}{c}[T(s_0)+\overline{B}(l_0)K_i{]}^T{P}_i+P_i[T(s_0)+\overline{B}(l_0)K_i]\hfill \\ +Q+K_i^T{K}_i=0\hfill \end{array}$$

(24)

Proof. Dividing by $\mathrm{\Delta }t$ on both sides of (22) and taking a limit, we attain

$$\begin{array}{c}0=\underset{\mathrm{\Delta }t\to 0}{lim}\frac{X^T(t+\mathrm{\Delta }t)P_{i}X(t+\mathrm{\Delta }t)-X^T(t)P_{i}X(t)}{\mathrm{\Delta }t}\hfill \\ +\underset{\mathrm{\Delta }t\to 0}{lim}\frac{{\int }_t^{t+\mathrm{\Delta }t}{X}^T(Q+K_i^T{K}_i)\mathrm{Xdt}}{\mathrm{\Delta }t}\hfill \\ =\frac{d{X}^T(t)P_{i}X(t)}{\mathrm{dt}}+\underset{\mathrm{\Delta }t\to 0}{lim}\frac{d}{d\mathrm{\Delta }t}{\int }_t^{t+\mathrm{\Delta }t}\hfill \\ \multicolumn{1}{c}{X^T(Q+K_i^T{K}_i)\mathrm{Xdt}}\\ =X^T[T(s_0)+\overline{B}(l_0)K_i{]}^T{P}_{i}X\hfill \\ \multicolumn{1}{c}{+X^T{P}_i[T(s_0)+\overline{B}(l_0)K_i]X}\\ +X^T(Q+K_i^T{K}_i)X.\hfill \end{array}$$

Thus, (22) implies (24).

Furthermore, take into account the asymptotically stable systems, $\stackrel{·}{X}=(T(s_0)-\overline{B}(l_0)K_i)X$ . Selecting the Lyapunov function as $V_i(X)=X^T{P}_{i}X$ , and taking its time derivative along the yields

$$\begin{array}{c}\frac{d}{\mathrm{dt}}(X^T{P}_{i}X)=X^T[(T(s_0)+\overline{B}(l_0)K_i{)}^T{P}_i\hfill \\ +P_i(T(s_0)+\overline{B}(l_0)K_i)]X.\hfill \end{array}$$

Calculating definite integral from t to $t+\mathrm{\Delta }t$ leads to

$$\begin{array}{c}{\int }_t^{t+\mathrm{\Delta }t}{X}^T(Q+K_i^T{K}_i)\mathrm{Xd}\tau =X^T(t+\mathrm{\Delta }t)P_{i}X(t+\mathrm{\Delta }t)\hfill \\ -X^T(t)P_{i}X(t),\hfill \end{array}$$

which is (22). Hence, the proof is complete.

Therefore, the integral reinforcement relation (22) is equivalent to equation (19) in Algorithm 1, which is an extension of Kleinman’s algorithm. This indicates the convergence of Algorithm 2.

Mismatched uncertain linear system

The robust tracking problem of a mismatched uncertain linear system (1) is discussed in this section. When the matched conditions (2) and (3) are not satisfied in system, we consider the design of robust tracking control law. The robust tracking problem is transformed into designing a robust control law for an augmented uncertain linear system. By calculating iteratively the optimal control law for an extended nominal system with a properly defined performance index, the online PI method is used to obtain robust tracking control law.

Obviously, the augmented system (9) is a mismatched uncertain linear system when the uncertain system (1) does not match conditions (2) and (3). In addition, the uncertainty of the augmented uncertain linear system can be decomposed into a matched part and a mismatched part according to (4) and (5):

$$\begin{array}{c}T(s)-T(s_0)=\left[\begin{array}{cc}A(s)-A(s_0)\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]\hfill \\ =\left[\begin{array}{cc}B(l_0)B{(l_0)}^{+}[A(s)-A(s_0)]\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]\hfill \\ +\left[\begin{array}{cc}[I-B(l_0)B{(l_0)}^{+}][A(s)-A(s_0)]\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right],\hfill \\ =\left[\begin{array}{cc}B(l_0)B{(l_0)}^{+}\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]\left[\begin{array}{cc}A(s)-A(s_0)\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]\hfill \\ +\left[\begin{array}{cc}I-B(l_0)B{(l_0)}^{+}\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]\left[\begin{array}{cc}A(s)-A(s_0)\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]\hfill \\ =\overline{B}(l_0)\overline{B}(l_0{)}^{+}[T(s)-T(s_0)]\hfill \\ +[\overline{I}-\overline{B}(l_0)\overline{B}(l_0{)}^{+}][T(s)-T(s_0)]\hfill \end{array}$$

and

$$\begin{array}{c}\overline{B}(l)-\overline{B}(l_0)=\overline{B}(l_0)\overline{B}(l_0{)}^{+}[\overline{B}(l)-\overline{B}(l_0)]\hfill \\ +[I-\overline{B}(l_0)\overline{B}(l_0{)}^{+}][\overline{B}(l)-\overline{B}(l_0)],\hfill \end{array}$$

where $\overline{B}(l_0)=\left[\begin{array}{c}B(l_0)\hfill \\ \multicolumn{1}{c}{O}\end{array}\right]$ , $\overline{B}(l_0{)}^{+}=\left[\begin{array}{cc}B{(l_0)}^{+}\hfill & O\hfill \end{array}\right]$ , $\overline{I}=\left[\begin{array}{cc}I\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]$ , and O is a zero matrix of appropriate dimensions.

• Lemma 3. If Assumptions 3 and 4 hold, then the uncertainty of the augmented uncertain linear system (9) satisfies the bounded conditions as follows:

$$[T(s)-T(s_0){]}^T[\overline{B}(l_0{)}^{+}{]}^T\overline{B}(l_0{)}^{+}[T(s)-T(s_0)]\le \overline{F}$$

(25)

$$[T(s)-T(s_0){]}^T[T(s)-T(s_0)]\le \overline{H},$$

(26)

where $\overline{F}=\left[\begin{array}{cc}F\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]$ , $\overline{H}=\left[\begin{array}{cc}H\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]$ .

Proof. Denote $\tilde{A}=A(s)-A(s_0)$ . From Assumption 3, we can obtain

$$\begin{array}{c}[T(s)-T(s_0){]}^T[\overline{B}(l_0{)}^{+}{]}^T\overline{B}(l_0{)}^{+}[T(s)-T(s_0)]\hfill \\ =\left[\begin{array}{cc}{\tilde{A}}^T\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]\left[\begin{array}{c}{[B{(l_0)}^{+}]}^T\hfill \\ \multicolumn{1}{c}{O}\end{array}\right]\hfill \\ \times \left[\begin{array}{cc}B{(l_0)}^{+}\hfill & O\hfill \end{array}\right]\left[\begin{array}{cc}\tilde{A}\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]\hfill \\ =\left[\begin{array}{cc}{\tilde{A}}^T{[B{(l_0)}^{+}]}^{T}B{(l_0)}^{+}\tilde{A}\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]\hfill \\ \le \left[\begin{array}{cc}F\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]\hfill \\ =\overline{F}.\hfill \end{array}$$

Therefore, (25) holds. In addition, we can achieve

$$\begin{array}{c}\multicolumn{1}{c}{[T(s)-T(s_0){]}^T[T(s)-T(s_0)]=\left[\begin{array}{cc}{\tilde{A}}^T\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]\left[\begin{array}{cc}\tilde{A}\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]}\\ \multicolumn{1}{c}{=\left[\begin{array}{cc}{\tilde{A}}^T\tilde{A}\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]}\\ \multicolumn{1}{c}{\le \left[\begin{array}{cc}H\hfill & O\hfill \\ \multicolumn{1}{c}{O}& O\hfill \end{array}\right]}\\ \multicolumn{1}{c}{=\overline{H}.}\end{array}$$

Therefore, (26) holds, thus completing the proof.

We construct the following optimal control problem. For an extended nominal linear system,

$$\stackrel{·}{X}=T(s_0)X+\overline{B}(l_0)u+(I-\overline{B}(l_0)\overline{B}(l_0{)}^{+})v,$$

(27)

find an augmented state feedback law, $u=\mathrm{KX},v=\mathrm{LX}$ , in order to minimize the performance index

$${\int }_0^{\infty }[X^T(\overline{F}+\overline{H}+{\beta }^{2}I)X+u^{T}u+v^{T}v]\mathrm{dt},$$

(28)

where $\beta \ge 0$ is a design parameter.

We denote $\tilde{B}=\left[\begin{array}{cc}\overline{B}(l_0)\hfill & (I-\overline{B}(l_0)\overline{B}{(l_0)}^{+})\hfill \end{array}\right]$ and $\overline{u}=\left[\begin{array}{c}u\hfill \\ \multicolumn{1}{c}{v}\end{array}\right]$ . Then rewriting (27) yields

$$\stackrel{·}{X}=T(s_0)X+\tilde{B}\overline{u},$$

and (28) can be rewritten as

$${\int }_0^{\infty }[X^T(\overline{F}+\overline{H}+{\beta }^{2}I)X+{\overline{u}}^T\overline{u}]\mathrm{dt}.$$

By computing the optimal control law, it follows that

$$\left[\begin{array}{c}u\hfill \\ \multicolumn{1}{c}{v}\end{array}\right]=-{\tilde{B}}^T\mathrm{PX}=\left[\begin{array}{c}-{\overline{B}}^T(l_0)P\hfill \\ \multicolumn{1}{c}{-{(I-\overline{B}(l_0)\overline{B}{(l_0)}^{+})}^{T}P}\end{array}\right]X,$$

(29)

where P is a positive definite matrix solving the following ARE

$$\mathrm{PT}(s_0)+T^T(s_0)P+\overline{F}+\overline{H}+{\beta }^{2}I-P\tilde{B}{\tilde{B}}^{T}P=0$$

(30)

The following theorem illustrates that optimal control law (29) can stabilize the mismatched uncertain linear system (9).

• Theorem 3. We assume that $u=\mathrm{KX}$ and $v=\mathrm{LX}$ are the solutions to the optimal control problems (27) and (28). If parameter $\beta$ can be chosen to hold the following conditions:

$$\begin{array}{c}{\beta }^{2}I-2L^{T}L>0,[\overline{B}(l)-\overline{B}(l_0)]{\overline{B}}^T(l_0)\ge 0,\hfill \end{array}$$

(31)

then the optimal control law, $u=\mathrm{KX}$ , with $K=-{\overline{B}}^T(l_0)P$ can stabilize the mismatched augmented uncertain system (9). That is, for every $s\in \mathcal{S},l\in \mathcal{L}$ , the mismatched closed-loop system

$$\stackrel{·}{X}=[T(s)+\overline{B}(l)K]X$$

(32)

is asymptotically stable.

Proof. Choosing the Lyapunov function as $V(X)=X^T\mathrm{PX}$ and taking its time derivative, along the closed-loop system (32), we obtain

$$\begin{array}{c}\multicolumn{1}{c}{\frac{\mathrm{dV}}{\mathrm{dt}}=X^T[(T^T(s)P+\mathrm{PT}(s)+2P\overline{B}(l)K)X}\\ \multicolumn{1}{c}{=X^T[(T(s)-T(s_0){)}^{T}P+P(T(s)-T(s_0))]X}\\ \multicolumn{1}{c}{+X^T[(T^T(s_0)P+\mathrm{PT}(s_0)]X}\\ \multicolumn{1}{c}{+2X^{T}P\overline{B}(l_0)K+2P(\overline{B}(l)-\overline{B}(l_0))K]X.}\end{array}$$

(33)

By decomposing the matched and mismatched parts in the augmented uncertain system (9), we will have

$$\begin{array}{c}\frac{\mathrm{dV}}{\mathrm{dt}}=X^T[T^T(s_0)P+\mathrm{PT}(s_0)]X\hfill \\ +X^T[T(s)-T(s_0){]}^T{\overline{B}}^T(l_0{)}^{+}{\overline{B}}^T(l_0)\mathrm{PX}\hfill \\ +X^T[T(s)-T(s_0){]}^T(I-\overline{B}(l_0)\overline{B}(l_0{)}^{+}{)}^{T}P]X\hfill \\ +X^{T}P\overline{B}(l_0)\overline{B}(l_0{)}^{+}(T(s)-T(s_0))X\hfill \\ +X^{T}P(I-\overline{B}(l_0)\overline{B}(l_0{)}^{+})(T(s)-T(s_0))X\hfill \\ +X^T[2P\overline{B}(l_0)K+2P(\overline{B}(l)-\overline{B}(l_0))K]X.\hfill \end{array}$$

(34)

According to the ARE (30)

$$\begin{array}{c}\frac{\mathrm{dV}}{\mathrm{dt}}=-X^T(\overline{F}+\overline{H}+{\beta }^{2}I)X+X^{T}P[\overline{B}(l_0){\overline{B}}^T(l_0)\hfill \\ +(I-\overline{B}(l_0)\overline{B}(l_0{)}^{+})(I-\overline{B}(l_0)\overline{B}(l_0{)}^{+}{)}^T]\mathrm{PX}\hfill \\ +X^T(T(s)-T(s_0){)}^T({\overline{B}}^T(l_0){)}^{+}{\overline{B}}^T(l_0)\mathrm{PX}\hfill \\ +X^T(T(s)-T(s_0){)}^T(I-\overline{B}(l_0)\overline{B}(l_0{)}^{+}{)}^T\mathrm{PX}\hfill \\ +X^{T}P\overline{B}(l_0)\overline{B}(l_0{)}^{+}(T(s)-T(s_0))X\hfill \\ +X^{T}P(I-\overline{B}(l_0)\overline{B}(l_0{)}^{+})(T(s)-T(s_0))X\hfill \\ +2X^{T}P\overline{B}(l_0)\mathrm{KX}+2x^{T}P(\overline{B}(l)-\overline{B}(l_0))\mathrm{KX}.\hfill \end{array}$$

By $K=-{\overline{B}}^T(l_0)P$ , we will have

$$\begin{array}{c}\frac{\mathrm{dV}}{\mathrm{dt}}=-X^T(\overline{F}+\overline{H}+{\beta }^{2}I)X+X^T{K}^T\mathrm{KX}\hfill \\ +X^{T}P(I-\overline{B}(l_0)\overline{B}(l_0{)}^{+})(I-\overline{B}(l_0)\overline{B}(l_0{)}^{+}{)}^T\mathrm{PX}\hfill \\ -X^T(T(s)-T(s_0){)}^T({\overline{B}}^T(l_0){)}^{+}\mathrm{KX}\hfill \\ +X^T(T(s)-T(s_0){)}^T(I-\overline{B}(l_0)\overline{B}(l_0{)}^{+}{)}^T\mathrm{PX}\hfill \\ -X^T{K}^T\overline{B}(l_0{)}^{+}(T(s)-T(s_0))X\hfill \\ +X^{T}P(I-\overline{B}(l_0)\overline{B}(l_0{)}^{+})(T(s)-T(s_0))X\hfill \\ -2X^T{K}^T\mathrm{KX}-2X^{T}P(\overline{B}(l)-\overline{B}(l_0)){\overline{B}}^T(l_0)\mathrm{PX}.\hfill \end{array}$$

By $L=-(I-\overline{B}(l_0)\overline{B}(l_0{)}^{+}{)}^{T}P$ , it follows that

$$\begin{array}{c}\frac{\mathrm{dV}}{\mathrm{dt}}=-X^T(\overline{F}+\overline{H}+{\beta }^{2}I)X-X^T{K}^T\mathrm{KX}+X^T{L}^T\mathrm{LX}\hfill \\ -2X^T{K}^T\overline{B}(l_0{)}^{+}(T(s)-T(s_0))X\hfill \\ -2X^T(T(s)-T(s_0){)}^T\mathrm{LX}\hfill \\ -2X^{T}P(\overline{B}(l)-\overline{B}(l_0)){\overline{B}}^T(l_0)\mathrm{PX}.\hfill \end{array}$$

Because

$$\begin{array}{c}-X^T{K}^T\mathrm{KX}-2X^T{K}^T\overline{B}(l_0{)}^{+}(T(s)-T(s_0))X\hfill \\ =-X^T[K-\overline{B}(l_0{)}^{+}(T(s)-T(s_0)){]}^T\hfill \\ \times [K-\overline{B}(l_0{)}^{+}(T(s)-T(s_0))]X\hfill \\ +X^T[\overline{B}(l_0{)}^{+}(T(s)-T(s_0)){]}^T\hfill \\ \times [\overline{B}(l_0{)}^{+}(T(s)-T(s_0))]X\hfill \\ \le X^T[\overline{B}(l_0{)}^{+}(T(s)-T(s_0)){]}^T\hfill \\ \times [\overline{B}(l_0{)}^{+}(T(s)-T(s_0))]X,\hfill \end{array}$$

and

$$\begin{array}{c}-2X^T{L}^T(T(s)-T(s_0))X\hfill \\ \le X^T(T(s)-T(s_0){)}^T(T(s)-T(s_0))X+X^T{L}^T\mathrm{LX}\hfill \\ \le X^T{L}^T\mathrm{LX}+X^T\mathrm{HX},\hfill \end{array}$$

hence

$$\begin{array}{c}\frac{\mathrm{dV}}{\mathrm{dt}}=-X^T(\overline{F}+\overline{H}+{\beta }^{2}I)X+X^T{L}^T\mathrm{LX}\hfill \\ -X^T{K}^T\mathrm{KX}-2X^T{K}^T\overline{B}(l_0{)}^{+}(T(s)-T(s_0))X\hfill \\ -2X^T(T(s)-T(s_0){)}^T\mathrm{LX}\hfill \\ -2X^{T}P(\overline{B}(l)-\overline{B}(l_0)){\overline{B}}^T(l_0)\mathrm{PX}\hfill \\ \le -X^T(\overline{F}+\overline{H}+{\beta }^{2}I)X+2X^T{L}^T\mathrm{LX}+X^T\overline{H}X\hfill \\ +X^T[\overline{B}(l_0{)}^{+}(T(s)-T(s_0)){]}^T\hfill \\ \times [\overline{B}(l_0{)}^{+}(T(s)-T(s_0))]X\hfill \\ -2X^{T}P(\overline{B}(l)-\overline{B}(l_0)){\overline{B}}^T(l_0)\mathrm{PX}\hfill \\ \le -2X^{T}P(\overline{B}(l)-\overline{B}(l_0)){\overline{B}}^T(l_0)\mathrm{PX}\hfill \\ -X^T({\beta }^{2}I-2L^{T}L)X.\hfill \end{array}$$

According to the conditions in (31)

$$\frac{\mathrm{dV}}{\mathrm{dt}}<0.$$

Therefore, the mismatched uncertain linear system (9) is asymptotically stable for every $s\in \mathcal{S}$ and $l\in \mathcal{L}$ ³¹ . That is, $u=-{\overline{B}}^T(l_0)\mathrm{PX}$ is a robust controller for the augmented mismatched uncertain system (9). Hence, the proof is complete.

• Remark 3. Generally, the pseudo-inverse of matrix, $\overline{B}(l_0{)}^{+}$ , will exist if its column vectors are not linearly dependent. ²⁹ In practical application, the input matrix, $\overline{B}(l_0)$ , is usually a matrix with column full-rank. Therefore, the pseudo-inverse of the input matrix $\overline{B}(l_0)$ is generally satisfied. Furthermore, the pseudo-inverse $\overline{B}(l_0{)}^{+}$ satisfies $\overline{B}(l_0{)}^{+}\overline{B}(l_0)=I$ . However, it does not satisfy $\overline{B}(l_0)\overline{B}(l_0{)}^{+}=I$ .

Here, we propose a robust tracking PI algorithm based on the RL for mismatched linear systems.

For any initial time, t, the optimal performance function (28) can be written as

$$\begin{array}{c}V(X(t))={\int }_t^{\infty }[X^T\overline{Q}X+{\overline{u}}^T\overline{u}]\mathrm{dt}\hfill \\ ={\int }_t^{t+\mathrm{\Delta }t}[X^T\overline{Q}X+{\overline{u}}^T\overline{u}]\mathrm{dt}\hfill \\ +{\int }_{t+\mathrm{\Delta }t}^{\infty }[X^T\overline{Q}X+{\overline{u}}^T\overline{u}]\mathrm{dt},\hfill \end{array}$$

where $\overline{Q}=\overline{F}+\overline{H}+{\beta }^{2}I$ .

Similar to Algorithm 2, the following online PI algorithm can be utilized to calculate the robust tracking law of a mismatched uncertain linear system.

• Algorithm 3. The online PI algorithm for robust tracking of mismatched uncertain linear systems

• Policy evaluation: Compute $P_i$ from

  $$\begin{array}{c}{X}^T(t)P_{i}X(t)={\int }_t^{t+\mathrm{\Delta }t}[X^T\overline{Q}x+X^T{\overline{K}}_i^T{\overline{K}}_{i}X]\mathrm{dt}\hfill \\ +X^T(t+\mathrm{\Delta }t)P_{i}X(t+\mathrm{\Delta }t).\hfill \end{array}$$

  (35)

• Policy improvement: Compute ${\overline{K}}_{i+1}$

$${\overline{K}}_{i+1}=-{\tilde{B}}^T{P}_i.$$

(36)

The convergence of the algorithm is proven similar to that of the matched system, which is omitted here.

• Remark 4. Robust tracking design is a challenging problem,^30,33 particularly when the signals to track is unstable. We consider the tracking problem of a class of polynomial signals that may be unstable. Algorithm 3 is an online robust control method, which can be available for mismatched uncertain systems with unknowing nominal state matrix. Using the least squares method, the matrix $P_i$ can be solved using (35) through the online trajectory data of system (27). We used the PI algorithm in the RL to solve this problem, which is a novel approach. In Theorem 3, it is clear that if parameter $\beta$ is selected to be large relatively, then the conditions in (31) easily matches in many practical applications.

Numerical experiments

The results of two numerical experiments are demonstrated to examine the viability of the theoretical frameworks. Moreover, the proposed PI algorithms are applied to solve the robust tracking of matched and mismatched uncertain systems effectively.

• Example 1. A matched uncertain linear system is considered as follows:

$$\begin{array}{c}\stackrel{·}{x}=\left[\begin{array}{cc}0\hfill & 2\hfill \\ \multicolumn{1}{c}{2+s}& s\hfill \end{array}\right]x+\left[\begin{array}{c}0\hfill \\ \multicolumn{1}{c}{l}\end{array}\right]u,\hfill \\ y=\left[\begin{array}{cc}1\hfill & 0\hfill \end{array}\right]x,\hfill \end{array}$$

(37)

where $s\in [0,3]$ and $l\in [1,3]$ are uncertain parameters. The reference signal is assumed to be $y_r=t+5$ . The control objectives of system is to design a state feedback law, $u=\mathrm{Kx}$ , which can make the system output $y=\mathrm{Cx}$ asymptotically track the reference signal for every $s\in [0,3]$ and $l\in [1,3]$ .

We denote $A(s)=\left[\begin{array}{cc}0\hfill & 2\hfill \\ \multicolumn{1}{c}{2+s}& s\hfill \end{array}\right],B(l)=\left[\begin{array}{c}0\hfill \\ \multicolumn{1}{c}{l}\end{array}\right]$ , and $C=[1,0]$ , respectively. Let $s_0=0$ and $l_0=1$ . According to (8), we will have

$$\begin{array}{cc}T(s)\hfill & =\left[\begin{array}{cccc}0\hfill & 2\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{2+s}& s\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{1}& 0\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 1\hfill & 0\hfill \end{array}\right],\overline{B}(l)=\left[\begin{array}{c}0\hfill \\ \multicolumn{1}{c}{l}\\ \multicolumn{1}{c}{0}\\ \multicolumn{1}{c}{0}\end{array}\right],\hfill \\ \multicolumn{1}{c}{N}& =\left[\begin{array}{c}0\hfill \\ \multicolumn{1}{c}{0}\\ \multicolumn{1}{c}{-1}\\ \multicolumn{1}{c}{0}\end{array}\right].\hfill \end{array}$$

$$\begin{array}{c}T(s)-T(s_0)=\left[\begin{array}{cccc}0\hfill & 0\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{s}& s\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill & 0\hfill \end{array}\right]\hfill \\ =\left[\begin{array}{c}0\hfill \\ \multicolumn{1}{c}{1}\\ \multicolumn{1}{c}{0}\\ \multicolumn{1}{c}{0}\end{array}\right]\left[\begin{array}{cccc}s\hfill & s\hfill & 0\hfill & 0\hfill \end{array}\right]\equiv \overline{B}(l_0)\mathrm{\Phi }(s),\hfill \end{array}$$

(38)

and

$$\begin{array}{c}\overline{B}(l)-\overline{B}(l_0)=\left[\begin{array}{c}0\hfill \\ \multicolumn{1}{c}{1}\\ \multicolumn{1}{c}{0}\\ \multicolumn{1}{c}{0}\end{array}\right](l-1)\equiv \overline{B}(l_0)\overline{\phi }(l),\hfill \end{array}$$

(39)

which implies that the parameters perturbation in system (37) are matched. Obviously, $\overline{\phi }(l)=(l-1)\ge 0$ for any $l\in [1,3]$ .

According to (38), we will have

$$\begin{array}{c}\multicolumn{1}{c}{{\mathrm{\Phi }}^T(s)\mathrm{\Phi }(s)=\left[\begin{array}{cccc}{s}^2\hfill & s^2\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{s^2}& s^2\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill & 0\hfill \end{array}\right]\le \left[\begin{array}{cccc}9\hfill & 9\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{9}& 9\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill & 0\hfill \end{array}\right]}\\ =\overline{M}.\hfill \end{array}$$

(40)

Therefore, the robust tracking problem is finally transformed into solving an optimal control of the nominal system. For the augmented nominal system,

$$\stackrel{·}{X}=\left[\begin{array}{cccc}0\hfill & 2\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{2}& 0\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{1}& 0\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 1\hfill & 0\hfill \end{array}\right]X+\left[\begin{array}{c}0\hfill \\ \multicolumn{1}{c}{1}\\ \multicolumn{1}{c}{0}\\ \multicolumn{1}{c}{0}\end{array}\right]u,$$

design a state feedback law, $u=\mathrm{KX}$ , in order to minimize the performance index

$$V(X(t_0),u(.))={\int }_{t_0}^{\infty }[X^T\mathrm{QX}+u^{T}u]\mathrm{dt},$$

(41)

where

$$Q=\overline{M}+I=\left[\begin{array}{cccc}10\hfill & 9\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{9}& 10\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 1\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill & 1.\hfill \end{array}\right]$$

(42)

By applying Algorithm 2, a robust tracking-control gain is obtained for system (37). We chose the initial control gain as $[-14.5-8-20-6]$ . The initial condition of the augmented nominal system was selected as $X_0=[0.200.150{]}^T$ . By using MATLAB, after four iterations, the positive definite matrix P and the control gain converge to the optimal solutions as follows:

$$P=\left[\begin{array}{cccc}13.4446\hfill & 8.0880\hfill & 11.4490\hfill & 3.2698\hfill \\ \multicolumn{1}{c}{8.0880}& 6.5092\hfill & 3.6855\hfill & 1.0071\hfill \\ \multicolumn{1}{c}{11.4490}& 3.6855\hfill & 18.9043\hfill & 6.0825\hfill \\ \multicolumn{1}{c}{3.2698}& 1.0071\hfill & 6.0825\hfill & 3.6346\hfill \end{array}\right],$$

and

$$K=\left[\begin{array}{cccc}-8.0880\hfill & -6.5092\hfill & -3.6855\hfill & -1.0071\hfill \end{array}\right].$$

We used MATLAB to solve the ARE directly, which is the implementation of Algorithm 1. Consequently, the following P matrix is obtained.

$$P=\left[\begin{array}{cccc}13.3014\hfill & 8.0402\hfill & 11.2417\hfill & 3.2466\hfill \\ \multicolumn{1}{c}{8.0402}& 6.4931\hfill & 3.6167\hfill & 1.0000\hfill \\ \multicolumn{1}{c}{11.2417}& 3.6167\hfill & 18.5987\hfill & 6.0402\hfill \\ \multicolumn{1}{c}{3.2466}& 1.0000\hfill & 6.0402\hfill & 3.6167\hfill \end{array}\right].$$

By comparison, the online PI algorithm has the same effect as the existing offline methods. But the online PI method does not need to know the nominal system matrix, which is the advantage of algorithm 2. It clearly shows that PI algorithm is effective in solving robust tracking problem. Figure 1 shows the evolution of the tracking-control signal. The convergence process of the elements in the P matrix is shown in Figure 2. It shows that optimal cost is obtained at time $t=4s$ after four updates of the controller parameters. The evolution of system output tracking reference signal with different parameters, s and l, are presented in Figure 3, which show that robust tracking is achieved for uncertain system (37).

• Example 2. A mismatched uncertain system is considered as follows:

$$\begin{array}{c}\stackrel{·}{x}=\left[\begin{array}{cc}s\hfill & s-1\hfill \\ \multicolumn{1}{c}{2}& -3\hfill \end{array}\right]x+\left[\begin{array}{c}0\hfill \\ \multicolumn{1}{c}{l}\end{array}\right]u,\hfill \\ y=\left[\begin{array}{cc}1\hfill & 0\hfill \end{array}\right]x,\hfill \end{array}$$

(43)

where $s\in [-2,2]$ and $l\in [1,3]$ are uncertain parameters. The reference signal is assumed to be $y_r=3$ . The control objectives of system is to design a state feedback law, $u=\mathrm{Kx}$ , which can make the system output $y=\mathrm{Cx}$ asymptotically track the reference signal for every $s\in [-2,2]$ and $l\in [1,3]$ .

Figure 1.

Control signal.

Figure 2.

P matrix iteration.

Figure 3.

Evolution of system output and reference signal with different parameters.

We denote $A(s)=\left[\begin{array}{cc}s\hfill & s-1\hfill \\ \multicolumn{1}{c}{2}& -3\hfill \end{array}\right],B(l)=\left[\begin{array}{c}0\hfill \\ \multicolumn{1}{c}{l}\end{array}\right]$ and $C=[1,0]$ , respectively. Let $s_0=0$ and $l_0=1$ . According to (8), we will have

$$\begin{array}{c}T(s)=\left[\begin{array}{ccc}s\hfill & s-1\hfill & 0\hfill \\ \multicolumn{1}{c}{2}& -3\hfill & 0\hfill \\ \multicolumn{1}{c}{1}& 0\hfill & 0\hfill \end{array}\right],\overline{B}(l)=\left[\begin{array}{c}0\hfill \\ \multicolumn{1}{c}{l}\\ \multicolumn{1}{c}{0}\end{array}\right],N=\left[\begin{array}{c}0\hfill \\ \multicolumn{1}{c}{0}\\ \multicolumn{1}{c}{-1}\end{array}\right].\hfill \end{array}$$

$$T(s)-T(s_0)=\left[\begin{array}{ccc}s\hfill & s\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& s\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill \end{array}\right],B(l)-B(l_0)=\left[\begin{array}{c}0\hfill \\ \multicolumn{1}{c}{l-1}\\ \multicolumn{1}{c}{0}\end{array}\right].$$

It is clear that the system is a mismatched system.

$$\overline{B}(l_0{)}^{+}=({\overline{B}}^T(l_0)\overline{B}(l_0){)}^{-1}{\overline{B}}^T(l_0)=\left[\begin{array}{ccc}0\hfill & 1\hfill & 0\hfill \end{array}\right]$$

$$I-\overline{B}(l_0)\overline{B}(l_0{)}^{+}=\left[\begin{array}{ccc}1\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 1\hfill \end{array}\right]$$

We chose $\beta =1$ . Denote $\mathrm{\Delta }T(s)=T(s)-T(s_0)$ . Therefore, we had

$$\begin{array}{c}\mathrm{\Delta }T{(s)}^T{(\overline{B}{(l_0)}^{+})}^T\overline{B}{(l_0)}^{+}\mathrm{\Delta }T(s)=\left[\begin{array}{ccc}0\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill \end{array}\right]=\overline{F},\hfill \end{array}$$

and

$$\begin{array}{c}\mathrm{\Delta }T{(s)}^T\mathrm{\Delta }T(s)=\left[\begin{array}{ccc}{s}^2\hfill & s^2\hfill & 0\hfill \\ \multicolumn{1}{c}{s^2}& s^2\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill \end{array}\right]\le \left[\begin{array}{ccc}4\hfill & 4\hfill & 0\hfill \\ \multicolumn{1}{c}{4}& 4\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill \end{array}\right]=\overline{H}.\hfill \end{array}$$

Therefore, the positive definite weight matrix $\overline{Q}$ is calculated as follows:

$$\overline{Q}=\overline{F}+\overline{H}+{\beta }^{2}I=\left[\begin{array}{ccc}5\hfill & 4\hfill & 0\hfill \\ \multicolumn{1}{c}{4}& 5\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 1\hfill \end{array}\right]$$

By applying Algorithm 3, a robust tracking-control gain is calculated for the mismatched uncertain system (43). The initial augmented control law is selected as

$$K=\left[\begin{array}{ccc}-2\hfill & -2\hfill & 0\hfill \\ \multicolumn{1}{c}{2}& -1\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill \\ \multicolumn{1}{c}{1}& 1\hfill & 8\hfill \end{array}\right].$$

The initial condition of the augmented nominal system is selected as $x_0=[3,-2,-8{]}^T$ . By using MATLAB, after four iterations, the positive definite matrix P and the augmented control gain converge to the optimal solutions as follows:

$$P=\left[\begin{array}{ccc}2.6301\hfill & 0.4074\hfill & 0.2535\hfill \\ \multicolumn{1}{c}{0.4074}& 0.6118\hfill & -0.0741\hfill \\ \multicolumn{1}{c}{0.2535}& -0.0741\hfill & 1.0090\hfill \end{array}\right]$$

and

$$K=\left[\begin{array}{ccc}-0.4074\hfill & -0.6118\hfill & 0.0741\hfill \\ \multicolumn{1}{c}{-2.6301}& -0.4074\hfill & -0.2535\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill \\ \multicolumn{1}{c}{-0.2535}& 0.0741\hfill & -1.0090\hfill \end{array}\right]$$

To solve the ARE directly using MATLAB software, the following P matrix is obtained:

$$P=\left[\begin{array}{ccc}2.6232\hfill & 0.4071\hfill & 0.2373\hfill \\ \multicolumn{1}{c}{0.4071}& 0.6076\hfill & -0.0730\hfill \\ \multicolumn{1}{c}{0.2373}& -0.0730\hfill & 0.9687\hfill \end{array}\right]$$

It clearly shows that PI algorithm is effective in solving robust tracking problem for mismatched uncertain system. Figure 4 shows the evolution of the tracking-control signal. The convergence process of the elements in the P matrix is shown in Figure 5. The evolution of the output and the reference trajectory with different parameters s and l are presented in Figure 6. As a result, a robust output tracking is achieved for mismatched uncertain system (43).

Figure 4.

Control signal.

Figure 5.

P matrix iteration.

Figure 6.

Evolution of system output and reference signal with different parameters.

Conclusion

In this study, RL-based online PI algorithms were proposed to calculate robust tracking control law for uncertain linear systems. It was based on an online policy iteration without using a nominal system matrix. The robust tracking problem was transformed into solving an optimal control with a predefined cost function. Based on the corresponding augmented linear system, offline and online PI algorithms were established to obtain a robust tracking controller. Numerical experiments were presented to demonstrate the effectiveness of the theoretical results. The proposed method may be developed to solve tracking problems of uncertain discrete-time systems, which may be the subject of our future research.