Design of LQ regulators with prescribed degree of stability for dual-rate systems based on reinforcement learning

Abstract

This paper considers dual-rate systems, where the output is measured at a relatively slow rate while the control signal is adjusted at a faster rate. The output sampling time is an integer multiple of the input sampling time. The paper examines dual-rate inferential control systems, which consist of a fast model, a slow model, and a switch. Missing output samples are estimated using the fast single-rate model. The single-rate control algorithm is then implemented at the fast-sampling rate. The fast-sampling discrete-time model is derived from the plant’s continuous-time model using the first-order hold (FOH) element. A discrete LQ regulator is proposed for this plant model, with a prescribed degree of stability (all closed-loop eigenvalues are within the range 0 < λ < 1 in magnitude). The matrix gain is calculated offline, and an online method for calculating the regulator gain is provided. The regulator gain is calculated using policy iteration, specifically Hewer’s algorithm. Finally, it is demonstrated that the presented inferential control system remains effective in the presence of multiplicative unmodeled dynamics. The main contributions of the paper are: (i) Designing the LQ regulator with a prescribed degree of stability using reinforcement learning (RL) (generalized policy iteration); and (ii) Considering the robust stability of the inferential control system in the presence of multiplicative unmodeled dynamics using the lifting technique.

Introduction

There are industrial processes that use digital control with sampled input and output values at different sample time intervals.^1,2 These are known as multi-rate systems. Standard control techniques cannot be used in these circumstances, leading to significant interest in these systems.^3,4

Kranc proposed the switch decomposition method for controlling multi-rate systems. This method involves transforming multi-rate systems (specifically dual-rate systems) into single-rate systems. The method has been further developed under the name “lifting technique,”^5–7 which is now a standard tool for transforming periodically time-varying systems into time-invariant ones. This technique is particularly important for state-space models.

Additionally, a dual-rate model that uses all available data (fast input and slow output data) can be derived using the polynomial transformation technique. ⁸ When stochastic disturbances have a non-Gaussian distribution, this method is used for the recursive identification of stochastic systems with unmodeled dynamics. ⁹ However, these transformation strategies require identifying more parameters. This issue was addressed by employing an accelerated stochastic approximation approach and the Bayesian information criteria, which allowed for identifying a fast model with fewer parameters. ¹⁰

In chemical processes, dual-rate systems are often used^11,12 and dual-rate techniques have recently been introduced in filtering theory. ¹³ Inferential control is an effective method for controlling dual-rate systems.^14,15 This method first estimates the missing output samples using a fast single-rate model. Then, it applies a single-rate regulator at the same fast sampling rate.

The paper assumes that the plant has a known continuous-time state-space model. The plant’s fast discrete-time model is derived using the first order hold (FOH) element. ¹⁶ A linear quadratic (LQ) regulator can be designed for this model. ¹⁷ The paper ¹⁸ proposes designing an LQ regulator, in the continuous time, to ensure that all poles of the closed-loop system lie in the left half-plane, Re{s} < −α, with α > 0 chosen by the designer. This approach provides greater tolerance for time delays and nonlinearities.

In this paper, we address the design of an LQ regulator with a specified degree of stability for discrete-time systems. The criterion ensures all closed-loop eigenvalues have magnitudes less than λ∈ (0,1]. The design procedure is typically completed offline.

We propose an online method for designing LQ regulators with a specified stability level, building on current active research ¹⁹ and documented in relevant monographs.^20–24 This iterative procedure solves the algebraic Riccati equation, forming the basis for LQ regulator gains, using generalized policy iteration derived from control theory principles.^25,26 Generalized policy iteration optimizes control laws iteratively until converging to optimal solutions for various dynamical systems and cost functions. Recursive feasibility, robust stability, and near-optimality properties are explored using policy iteration. ²⁷ Recent advancements in online policy iteration algorithms for optimal control in continuous-time systems with input constraints are discussed in. ²⁸ The intersection of reinforcement learning with adaptive control is explored in. ²⁹ Research references^30,31 address LQ regulator design under unknown linear system dynamics.

Reinforcement Learning (RL) is a broad area. Reference ³² explores RL based on differential games. Optimal control applications in industries using RL are discussed in Reference. ³³ Reference ³⁴ examines RL’s impact on decision-making under uncertainty. Reference ³⁵ describes a multi-agent system based on RL. Stochastic approximation algorithms and algorithms such as temporal-difference learning and Q-learning are detailed in Reference. ³⁶ Further developments in RL are also discussed.^36,37

The problem considered in this paper falls under model-based RL algorithms using adaptive dynamic programing. To the best of the authors’ knowledge, this problem has not been addressed in the literature.

The paper examines the robust stability of a closed-loop system controlled by the proposed LQ regulator, considering the presence of unmodeled dynamics in the form of multiplicative uncertainty. Through the use of the lifting technique, ³⁸ it demonstrates that dual-rate systems exhibit superior performance compared to fast-rate systems.

The main contributions of the paper are: (i) design of LQ regulators with the prescribed degree of stability for linear discrete-time fast rate model based on FOH and the design of reinforcement learning LQ regulators based on generalized policy iteration; (ii) consideration of robust stability of inferential systems in the presence multiplicative unmodeled dynamics using lifting techniques.

Problem formulation

Consider a single-input, single-output, single-rate system shown in Figure 1.

Figure 1.

The single-rate system.

In Figure 1. P _c represents a continuous linear time invariant (LTI) plant, H _h represents a zero-order hold (ZOH) and S _h is an ideal sampler. Both, H _h and S _h , operate with the sampling period h. Here, we introduce equivalent discrete time model for P _c

$$P=S_h{P}_c{H}_h$$

(1)

The standard discrete time control system is then shown in Figure 2.

Figure 2.

The discrete time single-rate control system.

In Figure 2, K represents a controller. In practical scenarios, sampling the output as fast as the input is often impossible due to physical sensor constraints. Therefore, in Figure 1, S _h is replaced with a slower sampler S _hp , where p ≥ 2 is an integer. The following figure illustrate this situation.

The input-output data are:

• (i) {u(kh): k = 0, 1, 2, ….} at a fast rate,

• (ii) {y(khp): k = 0, 1, 2, …} at a slow rate.

As a result, the intermediate output samples y(khp + j), for j = 1, 2, 3, …, p − 1, are not available. The dual-rate measurement is represented by the pair {u(kh), y(khp)}. The control system in this scenario is structured as shown in the figure below.

In Figure 4, $\hat{P}$ represents a model for the fast single-rate system P. The K is a fast single-rate regulator, and S denotes a switch. The fast output signal y _f (kh) comprises the slow-sampled output y(khp) taken every ph periods, along with the estimated output $\hat{y}(\mathrm{kh})$ from the model $\hat{P}$ .

It’s important to note that S _hp is equivalent to S _h followed by periodic switch S. Therefore, Figure 3 can be modified accordingly in the following figure.

Figure 3.

The dual-rate systems.

The signal $\hat{y}(\mathrm{kh})$ replaces the missing samples in y(khp). The feedback signal y_f (kh) is defined by the following relation:

$$y_f(\mathrm{kh})=\{\begin{array}{c}y\left(\mathrm{jph}\right),\mathrm{kh}=\mathrm{jph},j=0,1,2,3,\dots \hfill \\ \hat{y}\left((\mathrm{jp}+i)h\right),\mathrm{kh}=\mathrm{jph}+\mathrm{hi},i=0,1,2,3,...,p-1\hfill \end{array}$$

(2)

Finally, inferential control in Figure 4 consists of a fast-rate plant model, a fast single-rate regulator, and a periodic switch S.

Figure 4.

The sampled-data inferential control system.

From Figure 5, without unmodeled dynamics, it follows that $\hat{P}=P$ . This means the dual-rate system is equivalent to the single-rate system in Figure 2. Therefore, the LQ regulator design is based only on the fast-rate model. The presence of unmodeled dynamics will be considered later.

Figure 5.

Modified dual-rate system.

Fast model of the plant based on FOH method

Suppose the continuous time model of the plant P _c in state-space form is:

$$\stackrel{·}{x}\left(t\right)=A_{c}x\left(t\right)+B_{c}u\left(t\right)$$

(3)

In the next section, we will derive the discrete-time model $\hat{P}$ using the first-order hold (FOH) method. This method provides a more accurate discrete approximation of the continuous-time model by linearly extrapolating from current and previous input sequence elements.

$$\begin{array}{c}u\left(t\right)=u\left(\mathrm{kh}\right)+\frac{u\left(\mathrm{kh}\right)-u\left((k-1)h\right)}{h}(t-\mathrm{kh}),\hfill \\ \mathrm{kh}\le t\le (k+1)h\hfill \end{array}$$

(4)

By using instantaneous sampling, the sampler generates a discrete-time sequence:

$$z_u=z(\mathrm{kh})$$

(5)

The FOH dynamic is presented in the Figure 6.

Figure 6.

Impulse response of FOH.

The fast model $P_F=\hat{P}$ , based on the FOH approach, has the following form ¹⁶

$$\begin{array}{c}\multicolumn{1}{c}{\left[\begin{array}{c}x((k+1)h)\hfill \\ \multicolumn{1}{c}{u(\mathrm{kh})}\end{array}\right]=\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]\left[\begin{array}{c}x(\mathrm{kh})\hfill \\ \multicolumn{1}{c}{u((k-1)h)}\end{array}\right]+\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]u\left(\mathrm{kh}\right)}\\ =\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]x_p(\mathrm{kh})+\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]u\left(\mathrm{kh}\right)\hfill \end{array}$$

(6)

where

$$A=e^{A_{c}h}$$

$$B_1=\underset{0}{\overset{h}{\int }}(\frac{\eta }{h}-1)e^{A_c\eta }{B}_c\mathrm{d}\eta$$

$$B_2=\underset{0}{\overset{h}{\int }}(2-\frac{\eta }{h})e^{A_c\eta }{B}_c\mathrm{d}\eta$$

Finally, we can obtain the next block diagram for the FOH and system (6)

LQ with prescribed degree of stability for fast discrete time model

Now, we introduce a criterion for regulator design with constraints in equation (7). This criterion ensures that all closed-loop eigenvalues have magnitudes less than λ∈ (0,1], known as the closed-loop pole constraint. For system (6), the performance index is:

$$\left[\begin{array}{c}x\left((k+1)h\right)\hfill \\ \multicolumn{1}{c}{u\left(\mathrm{kh}\right)}\end{array}\right]=\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]\left[\begin{array}{c}x\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{u\left((k-1)h\right)}\end{array}\right]+\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]u\left(\mathrm{kh}\right)$$

$$\begin{array}{c}V\left(\left[\begin{array}{c}x\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{u\left((k-1)h\right)}\end{array}\right]\right)=\sum _{k=k_0}^{\infty }{\lambda }^{-\mathrm{zkh}}{\left[\begin{array}{c}x(\mathrm{kh})\hfill \\ \multicolumn{1}{c}{u((k-1)h)}\end{array}\right]}^T\hfill \\ Q\left[\begin{array}{c}x\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{u\left((k-1)h\right)}\end{array}\right]+u^T\left(\mathrm{kh}\right)\mathrm{Ru}(\mathrm{kh})\hfill \end{array}$$

(7)

The following theorem is now developed.

• Theorem 1. Let us suppose that for system (6) and criterion (7) is fulfilled.

• 1)  $(\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right],\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right])$ is completely controllable.

• 2) Let H be any matrix so that Q = HH^T. The pair $(\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right],H)$ is completely observable.

• 3)  $Q=Q^T\ge 0$ .

• 4)  $R=R^T>0$ .

• 5) Degree of stability for system (6) is λ∈ (0,1].

Then,

$$u\left(\mathrm{kh}\right)=-K\left[\begin{array}{c}x\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{u\left((k-1)h\right)}\end{array}\right]$$

where:

$$K={\lambda }^{-2h}{(R+{\lambda }^{-2h}{\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]}^{T}P\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right])}^{-1}{\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]}^{T}P\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]$$

and matrix P is a solution of the next algebraic Riccati equation:

$$\begin{array}{c}{({\lambda }^{-h}\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]-{\lambda }^{-h}\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]K)}^T\mathrm{P}({\lambda }^{-h}\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]-{\lambda }^{-h}\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]K)\hfill \\ -P+Q+K^T\mathrm{RK}=0\hfill \end{array}$$

• Proof: Let us introduce

$${\lambda }^{-\mathrm{kh}}\left[\begin{array}{c}x\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{u\left((k-1)h\right)}\end{array}\right]=\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}\left((k-1)h\right)}\end{array}\right]$$

(8)

$$\hat{u}\left(\mathrm{kh}\right)={\lambda }^{-\mathrm{kh}}u\left(\mathrm{kh}\right)$$

(9)

From relations (7)–(9) follows that:

$$\begin{array}{c}\hat{V}\left(\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}\left((k-1)h\right)}\end{array}\right]\right)=\sum _{k=k_0}^{\infty }{\lambda }^{-2\mathrm{kh}}{\left[\begin{array}{c}\hat{x}(\mathrm{kh})\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}((k-1)h)}\end{array}\right]}^T\hfill \\ Q\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}\left((k-1)h\right)}\end{array}\right]+{\hat{u}}^T\left(\mathrm{kh}\right)R\hat{u}(\mathrm{kh})\hfill \end{array}$$

(10)

From equations (6), (8), and (9) we have:

$$\begin{array}{c}\multicolumn{1}{c}{\left[\begin{array}{c}\hat{x}((k+1)h)\hfill \\ \multicolumn{1}{c}{\hat{u}\left(\mathrm{kh}\right)}\end{array}\right]={\lambda }^{-h}\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}\left((k-1)h\right)}\end{array}\right]}\\ +{\lambda }^{-h}\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]\hat{u}\left(\mathrm{kh}\right)\hfill \end{array}$$

(11)

owing the fact:

$${\lambda }^{-(k+1)h}\left[\begin{array}{c}x((k+1)h)\hfill \\ \multicolumn{1}{c}{u\left(\mathrm{kh}\right)}\end{array}\right]=\left[\begin{array}{c}\hat{x}((k+1)h)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}\left(\mathrm{kh}\right)}\end{array}\right]$$

(12)

The optimal performance index has a form:

$$\hat{V}(·)={\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}((k-1)h)}\end{array}\right]}^T\mathrm{P}\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}((k-1)h)}\end{array}\right]$$

(13)

where P is symmetric matrix.

The Bellman equation for our case is:

$$\begin{array}{c}{\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}((k-1)h)}\end{array}\right]}^T\mathrm{P}\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}((k-1)h)}\end{array}\right]={\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}((k-1)h)}\end{array}\right]}^T\hfill \\ Q\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}((k-1)h)}\end{array}\right]+{\hat{u}}^T\left(\mathrm{kh}\right)R\hat{u}\left(\mathrm{kh}\right)+{\left[\begin{array}{c}\hat{x}((k+1)h)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}\left(\mathrm{kh}\right)}\end{array}\right]}^T\hfill \\ \mathrm{P}\left[\begin{array}{c}\hat{x}((k+1)h)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}\left(\mathrm{kh}\right)}\end{array}\right]\hfill \end{array}$$

(14)

For last term in relation (14), by using relation (11), one can get:

$$\begin{array}{c}{({\lambda }^{-h}\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}\left((k-1)h\right)}\end{array}\right]+{\lambda }^{-h}\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]\hat{u}\left(\mathrm{kh}\right))}^T\hfill \\ P({\lambda }^{-h}\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}\left((k-1)h\right)}\end{array}\right]+{\lambda }^{-h}\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]\hat{u}\left(\mathrm{kh}\right))\hfill \\ ={\lambda }^{-2h}{\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}u\left((k-1)h\right)}\end{array}\right]}^T{\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]}^{T}P\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]\hfill \\ \left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}u\left((k-1)h\right)}\end{array}\right]+{\lambda }^{-2h}{\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}u\left((k-1)h\right)}\end{array}\right]}^T{\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]}^T\hfill \\ P\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]\hat{u}\left(\mathrm{kh}\right)+{\lambda }^{-2h}{\hat{u}}^T\left(\mathrm{kh}\right){\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]}^{T}P\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]\hat{u}\left(\mathrm{kh}\right)\hfill \end{array}$$

(15)

According to (13) we have:

$$\begin{array}{c}\hat{V}\left(\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}\left((k-1)h\right)}\end{array}\right]\right)={\left[\begin{array}{c}\hat{x}(\mathrm{kh})\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}((k-1)h)}\end{array}\right]}^T\hfill \\ P\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}\left((k-1)h\right)}\end{array}\right]\hfill \end{array}$$

(16)

From (14), (15) and (16) it follows that from:

$$\frac{\partial \hat{V}\left(\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}\left((k-1)h\right)}\end{array}\right]\right)}{\partial \hat{u}\left(\mathrm{kh}\right)}=0$$

(17)

we have:

$$\begin{array}{c}\hat{u}\left(\mathrm{kh}\right)=-{\lambda }^{-2h}{(R+{\lambda }^{-2h}{\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]}^{T}P\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right])}^{-1}{\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]}^T\hfill \\ P\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}\left((k-1)h\right)}\end{array}\right]=-K\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}\left((k-1)h\right)}\end{array}\right]\hfill \end{array}$$

(18)

where:

$$K={\lambda }^{-2h}{(R+{\lambda }^{-2h}{\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]}^{T}P\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right])}^{-1}{\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]}^{T}P\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]$$

(19)

By using (14) and (18) one can get equality:

$${\left[\begin{array}{c}\hat{x}(\mathrm{kh})\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}((k-1)h)}\end{array}\right]}^{T}T\left[\begin{array}{c}\hat{x}\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{{\lambda }^{-h}\hat{u}\left((k-1)h\right)}\end{array}\right]=0$$

(20)

Since this must hold for all state the matrix:

$$T=0$$

(21)

In our case, matrix T has a form:

$$\begin{array}{c}{({\lambda }^{-h}\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]-{\lambda }^{-h}\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]K)}^T\mathrm{P}({\lambda }^{-h}\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]-{\lambda }^{-h}\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]K)\hfill \\ -P+Q+K^T\mathrm{RK}=0\hfill \end{array}$$

(22)

If we put (19) into (22) after arranging the formula one can get:

$$\begin{array}{c}{\lambda }^{-2h}{\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]}^{T}P\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]-P+Q-{\lambda }^{-4h}{\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]}^T\hfill \\ P\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]·{(R+{\lambda }^{-2h}{\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]}^{T}P)}^{-1}{\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]}^{T}P\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]=0\hfill \end{array}$$

(23)

Based on relations (8), (9), (18) and (23) it follows the proof of the theorem.

LQ regulator design is described by Algorithm 1.

Algorithm 1

1. Choose the matrices $Q=Q^T\ge 0$ , $R=R^T>0$ , λ, and h.

2. For P, solve algebraic Riccati equation (23).

3. Determine the regulator’s gain K (relation (19)).

It is possible to see that the procedure for regulator design is off-line.

Robust stability of dual-rate systems

In this section, we consider robust stability of the dual-rate system. It is supposed that Assumption 1 is not valid. We treat $\hat{P}$ as a nominal model and consider multiplicative uncertainty. The uncertainty class is given by ³⁹ :

$$P_{\mathrm{\Delta }}\left(z\right)=\hat{P}(z)(I+W_1\left(z\right)\mathrm{\Delta }(z)W_2\left(z\right))$$

(24)

where $\mathrm{\Delta }(z)$ is a perturbation and $W_1\left(z\right)$ and $W_2\left(z\right)$ are fixed frequency weighting filters. Let us note that sampling period for input signal is h and for state signal is ph.

The standard technique for analysis and design of multi-rate systems is lifting. ³⁸ The lifting technique transforms periodically time-varying systems into time-invariant systems. Let u(kh) be a discrete-time signal defined on the set {0, 1, 2, …}.

$$u\left(\mathrm{kh}\right)=\{u\left(0\right),u\left(h\right),u\left(2h\right),\dots .,u\left(\mathrm{kh}\right),\dots \}$$

The lifting operator L is the map from $u\left(\mathrm{kh}\right)$ to $\underset{¯}{u}\left(\mathrm{kh}\right)$

$$\mathrm{L}:u\left(\mathrm{kh}\right)\to \underset{¯}{u}\left(\mathrm{kh}\right)$$

(25)

where:

$$\begin{array}{c}\underset{¯}{u}\left(\mathrm{kh}\right)=\hfill \\ \left\{\begin{array}{ccc}\left[\begin{array}{c}u(0)\hfill \\ \multicolumn{1}{c}{u(h)}\\ \multicolumn{1}{c}{\begin{array}{c}⋮\hfill \\ \multicolumn{1}{c}{u(\mathrm{ph}-1)}\end{array}}\end{array}\right],\hfill & \left[\begin{array}{c}u(\mathrm{ph})\hfill \\ \multicolumn{1}{c}{u(\mathrm{ph}+1)}\\ \multicolumn{1}{c}{\begin{array}{c}⋮\hfill \\ \multicolumn{1}{c}{u(2\mathrm{ph}-1)}\end{array}}\end{array}\right],\hfill & \begin{array}{cc}\dots \hfill & \left[\begin{array}{c}u(\mathrm{khp})\hfill \\ \multicolumn{1}{c}{u(\mathrm{khp}+1)}\\ \multicolumn{1}{c}{\begin{array}{c}⋮\hfill \\ \multicolumn{1}{c}{u(\mathrm{khp}+\mathrm{ph}-1)}\end{array}}\end{array}\right],\dots \hfill \end{array}\hfill \end{array}\right\}\hfill \end{array}$$

(26)

We now can formulate theorem for lifted systems.

• Theorem 2. Consider the system (6). Suppose the following:

• 1)  $T_1=h$ is the sampling period for input $\{u(·)\}$ .

• 2)  $T_2=\mathrm{ph},p>1$ is the sampling period for state $\left\{x_p\right\}$ .

Then,

• A. Lifted system for system (6) is:

$$x_p\left((k+1)\mathrm{ph}\right)=\underset{¯}{A}{x}_p\left(\mathrm{kph}\right)+\underset{¯}{B}\underset{¯}{u}(\mathrm{kph})$$

where:

$$\begin{array}{c}\multicolumn{1}{c}{x_p\left((k+1)\mathrm{ph}\right)=\left[\begin{array}{c}x((k+1)\mathrm{ph})\hfill \\ \multicolumn{1}{c}{u(\mathrm{kph})}\end{array}\right],\underset{¯}{u}\left(\mathrm{kph}\right)}\\ =\left[\begin{array}{c}u(\mathrm{kph})\hfill \\ \multicolumn{1}{c}{u(\mathrm{kph}+h)}\\ \multicolumn{1}{c}{\begin{array}{c}⋮\hfill \\ \multicolumn{1}{c}{u(\mathrm{kph}+\mathrm{ph}-1)}\end{array}}\end{array}\right]\hfill \end{array}$$

$$\begin{array}{c}\underset{¯}{A}=A_F^p,\underset{¯}{B}=[A_F^{p-1}{B}_F,A_F^{p-2}{B}_{F,\dots .}{B}_F],A_F=\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right],\hfill \\ B_F=\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{\mathrm{I}}\end{array}\right]\hfill \end{array}$$

• B. The lifted regulator has a gain:

$$\underset{¯}{K}={\lambda }^{-2\mathrm{ph}}{(R+{\lambda }^{-2\mathrm{ph}}\underset{¯}{B^T}\underset{¯}{P}\underset{¯}{B})}^{-1}\underset{¯}{B^T}\underset{¯}{P}\underset{¯}{A}$$

• Proof: It is possible to rewrite relation (6) in the next form:

$$x_p\left((k+1)h\right)=A_F{x}_p\left(\mathrm{kh}\right)+B_{F}u(\mathrm{kh})$$

Let us replace k with $\mathrm{kp}$ . We have:

$$\begin{array}{c}\multicolumn{1}{c}{x_p(\mathrm{kph}+h)=A_F{x}_p\left(\mathrm{kph}\right)+B_{F}u(\mathrm{kph})}\\ \multicolumn{1}{c}{x_p(\mathrm{kph}+2h)=A_F{x}_p(\mathrm{kph}+h)+B_{F}u(\mathrm{kph}+h)=}\\ =A_F(A_F{x}_p\left(\mathrm{kph}\right)+B_{F}u\left(\mathrm{kph}\right))+\mathrm{Bu}(\mathrm{kph}+h)=\hfill \\ =A_F^2{x}_p\left(\mathrm{kph}\right)+A_F{B}_{F}u\left(\mathrm{kph}\right)+B_{F}u(\mathrm{kph}+h)\hfill \\ \multicolumn{1}{c}{x_p(\mathrm{kph}+3h)=A_F{x}_p(\mathrm{kph}+2h)+B_{F}u(\mathrm{kph}+2h)=\dots =}\\ =A_F^3{x}_p\left(\mathrm{kph}\right)+A_F^2{B}_{F}u\left(\mathrm{kph}\right)+B_{F}u(\mathrm{kph}+2h)\hfill \\ ⋮\hfill \\ \multicolumn{1}{c}{x_p(\mathrm{kph}+\mathrm{ph})=A_F^p{x}_p\left(\mathrm{kph}\right)+A_F^{p-1}{B}_{F}u\left(\mathrm{kph}\right)}\\ +A_F^{p-2}{B}_{F}u(\mathrm{kph}+h)+\dots +B_{F}u(\mathrm{kph}+\mathrm{ph}-1)\hfill \end{array}$$

(27)

This equation we can rewrite in the next form:

$$\begin{array}{c}{x}_p(\mathrm{kph}+\mathrm{ph})=A_F^p{x}_p\left(\mathrm{kph}\right)+\begin{array}{ccc}{A}_F^{p-1}{B}_F\hfill & A_F^{p-2}{B}_F\hfill & \begin{array}{cc}\dots \hfill & B_F\hfill \end{array}\hfill \end{array}\hfill \\ \left[\begin{array}{c}u\left(\mathrm{kph}\right)\hfill \\ \multicolumn{1}{c}{u(\mathrm{kph}+h)}\\ \multicolumn{1}{c}{\begin{array}{c}⋮\hfill \\ \multicolumn{1}{c}{u(\mathrm{kph}+\mathrm{ph}-1)}\end{array}}\end{array}\right]=\underset{¯}{A}{x}_p\left(\mathrm{kph}\right)+\underset{¯}{B}\underset{¯}{u}(\mathrm{kph})\hfill \end{array}$$

(28)

The statement A) is proven. The results of statement B) follows from the solution of Riccati equation when we replace matrices A and B with matrices $\underset{¯}{A}$ and $\underset{¯}{B}$ . For the criterion for LQ regulator design, we use:

$${\lambda }^{-\mathrm{khp}}\left[\begin{array}{c}x(\mathrm{kph})\hfill \\ \multicolumn{1}{c}{u((k-1)\mathrm{ph})}\end{array}\right]=\left[\begin{array}{c}\hat{x}(\mathrm{kph})\hfill \\ \multicolumn{1}{c}{{\lambda }^{-\mathrm{ph}}\hat{u}((k-1)\mathrm{ph})}\end{array}\right]$$

(29)

$$\hat{u}\left(\mathrm{kph}\right)={\lambda }^{-\mathrm{kph}}u(\mathrm{kph})$$

(30)

Theorem is proved. According to Reference, ³⁸ we have the next lifted systems:

$${\underset{¯}{W}}_1\left(z\right)=L{W}_1(z)L^{-1},{\underset{¯}{W}}_2\left(z\right)=L{W}_2(z)L^{-1},\underset{¯}{\mathrm{\Delta }}\left(z\right)=L\underset{¯}{\mathrm{\Delta }}\left(z\right)L^{-1}$$

(31)

Lastly, we shall determine the lifted transfer function $\underset{¯}{\hat{P}}\left(z\right)$ ,

$$X_p\left(z\right)=\underset{¯}{\hat{P}}\left(z\right)U(z)$$

(32)

From relations (28) and (32), it follows:

$$\underset{¯}{\hat{P}}\left(z\right)={(\mathrm{zI}-\underset{¯}{A})}^{-1}\underset{¯}{B}$$

(33)

Now we present the model for switch S in Figure 7. According to Reference, ¹⁵ we have the situation as shown in the next figure:

Figure 7.

The sampled-data inferential system, H_F,h is FOH DA converter, $\hat{P}$ = P_F is fast plant model.

The R ₁ and R ₂ are static systems with the following matrix form:

$$R_1={\left[\begin{array}{ccc}\begin{array}{cc}1\hfill & 0\hfill \end{array}\hfill & \cdots \hfill & 0\hfill \\ \multicolumn{1}{c}{\begin{array}{cc}0\hfill & 0\hfill \end{array}}& \cdots \hfill & 0\hfill \\ \multicolumn{1}{c}{\begin{array}{cc}\begin{array}{c}⋮\hfill \\ \multicolumn{1}{c}{0}\end{array}\hfill & \begin{array}{c}⋮\hfill \\ \multicolumn{1}{c}{0}\end{array}\hfill \end{array}}& \begin{array}{c}\ddots \hfill \\ \multicolumn{1}{c}{\cdots }\end{array}\hfill & \begin{array}{c}⋮\hfill \\ \multicolumn{1}{c}{0}\end{array}\hfill \end{array}\right]}_{p\times p}$$

(34)

$$R_2={\left[\begin{array}{ccc}\begin{array}{cc}0\hfill & 0\hfill \end{array}\hfill & \cdots \hfill & 0\hfill \\ \multicolumn{1}{c}{\begin{array}{cc}0\hfill & 1\hfill \end{array}}& \cdots \hfill & 0\hfill \\ \multicolumn{1}{c}{\begin{array}{cc}\begin{array}{c}⋮\hfill \\ \multicolumn{1}{c}{0}\end{array}\hfill & \begin{array}{c}⋮\hfill \\ \multicolumn{1}{c}{0}\end{array}\hfill \end{array}}& \begin{array}{c}\ddots \hfill \\ \multicolumn{1}{c}{\cdots }\end{array}\hfill & \begin{array}{c}⋮\hfill \\ \multicolumn{1}{c}{1}\end{array}\hfill \end{array}\right]}_{p\times p}$$

(35)

The feedback signal $x_F(·)$ is defined as:

$$x_F(·)=R_1{x}_p(·)+R_2{\hat{x}}_p(·)$$

(36)

Let us notice that:

$$R_1+R_2=J$$

(37)

According to Assumption 1, along with relations (24), (34)–(37) and Figures 1 and 8, the following figure can be derived:

Figure 8.

The model of the switch in the sampled data inferential system.

Our main goal is to study the robust stability of dual-rate system in Figure 9. To apply the small gain theorem, ³⁹ we will convert system from Figure 9 to $M-\mathrm{\Delta }$ form ³⁹ as shown in Figure 10.

Figure 9.

The lifted inferential control system with multiplicative uncertainty.

Figure 10.

The $M-\mathrm{\Delta }$ structure for system in Figure 9.

Now let’s determine the matrix $\underset{¯}{M}$ . This is formulated as a theorem.

• Theorem 3. Let us consider the system presented in Figure 9. Suppose the following assumptions hold:

• 1) The perturbation $\underset{¯}{\mathrm{\Delta }}(z)$ is a stable and linear time-invariant system with norm $\underset{¯}{\mathrm{\Delta }}(z{)}_{\infty }<1$

• 2)  ${\underset{¯}{W}}_1\left(z\right)$ and ${\underset{¯}{W}}_2\left(z\right)$ are fixed frequency weighting filters that are stable and linear-time invariant systems.

Then, the matrix $\underset{¯}{M}(z)$ takes the form:

$$\underset{¯}{M}\left(z\right)=-{\underset{¯}{W}}_2\left(z\right){(I+\underset{¯}{K}\underset{¯}{\hat{P}}\left(z\right))}^{-1}\underset{¯}{K}{R}_1{\underset{¯}{W}}_1\left(z\right)\underset{¯}{\hat{P}}\left(z\right)$$

• Proof: From Figure 9 we derive the expression as follows:

$${\underset{¯}{U}}_{\mathrm{\Delta }}(z)={\underset{¯}{W}}_2\left(z\right)\underset{¯}{U}(z)$$

(38)

$$\underset{¯}{U}\left(z\right)=-\underset{¯}{K}{X}_F(z)$$

(39)

It is also noted that:

$${\hat{X}}_p\left(z\right)=-\underset{¯}{\hat{P}}(z)\underset{¯}{U}(z)$$

(40)

$${\hat{X}}_p\left(z\right)={\underset{¯}{W}}_1\left(z\right)\hat{P}\left(z\right)X_{p\mathrm{\Delta }}(z)$$

(41)

Using relation (37), we obtain for the feedback signal $X_F(z)$ :

$$\begin{array}{c}\multicolumn{1}{c}{X_F\left(z\right)=R_1{\hat{X}}_p\left(z\right)+(R_1+R_2)\underset{¯}{\hat{P}}\left(z\right)\underset{¯}{U}\left(z\right)}\\ =R_1{\underset{¯}{W}}_1\left(z\right)\underset{¯}{\hat{P}}\left(z\right)X_{p\mathrm{\Delta }}\left(z\right)+\underset{¯}{\hat{P}}\left(z\right)\underset{¯}{U}\left(z\right)\hfill \end{array}$$

(42)

Based on relation (39), it follows that:

$$\underset{¯}{U}\left(z\right)=-\underset{¯}{K}{R}_1{\underset{¯}{W}}_1\left(z\right)\underset{¯}{\hat{P}}\left(z\right)X_{p\mathrm{\Delta }}\left(z\right)-\underset{¯}{K}\underset{¯}{\hat{P}}\left(z\right)U\left(z\right)$$

(43)

Then we have:

$$\underset{¯}{U}\left(z\right)=-{(I+\underset{¯}{K}\underset{¯}{\hat{P}}\left(z\right))}^{-1}\underset{¯}{K}{R}_1{\underset{¯}{W}}_1\left(z\right)\underset{¯}{\hat{P}}\left(z\right)X_{p\mathrm{\Delta }}\left(z\right)$$

(44)

By using relation (39) and (43), it follows that:

$${\underset{¯}{U}}_{\mathrm{\Delta }}(z)=-{\underset{¯}{W}}_2\left(z\right){(I+\underset{¯}{K}\underset{¯}{\hat{P}}\left(z\right))}^{-1}\underset{¯}{K}{R}_1{\underset{¯}{W}}_1\left(z\right)\underset{¯}{\hat{P}}\left(z\right)X_{p\mathrm{\Delta }}\left(z\right)$$

(45)

The theorem is thus proved. Finally, we formulate a theorem for robust stability for the dual-rate system.

• Theorem 4. Suppose that the assumptions of Theorem 3 are satisfied, along with the following assumption:

• 1) System is nominally stable (K stabilizes $\hat{P}\left(z\right)$ ).

Then,

• A. For the dual-rate system to be stable for all admissible perturbation $\mathrm{\Delta }$ , it is sufficient that:

$$\Vert {\underset{¯}{W}}_2\left(z\right){(I+\underset{¯}{K}\underset{¯}{\hat{P}}\left(z\right))}^{-1}\underset{¯}{K}{R}_1{\underset{¯}{W}}_1\left(z\right)\underset{¯}{\hat{P}}\left(z\right){\Vert }_{\infty }<1$$

• B. The fast single-rate control is no more robust than the dual-rate inferential system.

• Proof: The proof is based on small gain theorem and manipulation with matrix norms, similar to Proposition 2 and Corollary 1 in Reference. ¹⁵

Model-based policy iteration algorithm for LQ regulator design

In this section, we discuss an online approach for designing regulators, using Hewer’s algorithm to solve the discrete-time Riccati equation. ²⁵ This method is rooted in reinforcement learning. We show that Hewer’s algorithm converges under stability and detectability assumptions.

Reinforcement learning suggests generalized policy iteration, ¹⁹ where the algorithm involves iterating l steps to solve the matrix equation in each iteration j. When l = 1, it corresponds to value iteration, and for l = ∞, it represents policy iteration. The algorithm, based on equations (19) and (22), is summarized in the table below.

Algorithm 2

1. Select matrices $Q=Q^T\ge 0$ , $R=R^T\ge 0$ , K ₀ (not necessarily stabilizing),

$$P_o=\mathrm{I},\epsilon >0,\lambda ,h>0,j=0,i=0,1,2,\dots l-1.$$

•  2. $P_j^0=P_j$

$$P_{j+1}=P_j^l$$

$$\begin{array}{c}{P}_j^{i+1}={({\lambda }^{-h}\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]-{\lambda }^{-h}\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]K_j)}^T\hfill \\ P_j^i({\lambda }^{-h}\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]-{\lambda }^{-h}\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]K_j)+Q+K_j^{T}R{K}_j\hfill \end{array}$$

•  3. $K_{j+1}={\lambda }^{-2h}{(R+{\lambda }^{-2h}{\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]}^T{P}_{j+1}\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right])}^{-1}{\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]}^T{P}_{j+1}\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]$ .

•  4. Stop if $\Vert K_{j+1}-K_j\Vert <\epsilon$ ,

Otherwise set j = j+1 and return to step 2.

• Remark 1. Theorem 1 explains how to design an LQ regulator for fast-rate models using the FOH element. This forms the basis for designing a reinforcement learning LQ regulator. Theorem 2 addresses the design of lifted systems, a common technique for multirate systems. The main outcome of Theorem 3 is the establishment of the M-Δ structure, which is crucial in robust control theory. Lastly, Theorem 4 deals with robust stability, demonstrating that fast-rate control systems are not more robust than dual-rate systems.

• Remark 2. The matrices Q and R are weighting matrices where Q is semi-positive definite (Q ≥ 0) and R is positive definite ( $\mathrm{R}>0$ ). These matrices are typically chosen through a trial-and-error process. When aiming for smaller squared errors, larger values are assigned to the corresponding diagonal elements in matrix Q.

In practical applications, choosing a smaller R speeds up the closed-loop response, while a larger R slows it down. More formal methods for choosing these matrices are detailed in Reference [17, Ch. 6].

The sampling rate must be high relative to the rate of changes in the signal being considered. A common rule of thumb is to ensure the sampling rate is 5–10 times the bandwidth of the system. It’s also common practice to use an analog filter before the sampling process. The sampling rate is equal to the sampling period of fast-rate systems.

The parameter λ∈ (0,1] is crucial; a smaller λ leads to a higher speed of convergence of states.

• Remark 3. This paper explores multirate (dual-rate) discrete-time systems using the FOH element for holding. The designed D/A converter achieves higher accuracy compared to ZOH. Multirate systems are important both theoretically and practically in discrete-time systems.

The first key finding, without unmodeled dynamics, shows that a dual-rate inferential system is equivalent to a single-rate (fast-rate) system. A corresponding LQ regulator with prescribed degree of stability ensures that all closed-loop poles lie within the λ-circle in the complex plane where $\mathrm{\lambda }\le 1$ . The regulator design process is offline and uses dynamic programing. For single-rate systems without unmodeled dynamics, a reinforcement learning (RL) LQ regulator is proposed, based on adaptive dynamic programing, with an online design process. The ultimate aim is to develop a model-free RL LQ controller.

The second key result addresses robust stability in the presence of unmodeled dynamics for LQ controllers. This involves transforming the dual-rate system into a single-rate system (lifted model) and establishing the $\mathrm{M}-\mathrm{\Delta }$ model crucial in robust control theory. It demonstrates system stability for all allowable perturbations $\mathrm{\Delta }$ , ensuring $\Vert \mathrm{M}{\Vert }_{\infty }<1$ . A significant finding is that a fast single-rate system is no more robust than dual-rate inferential systems. This result can be extended to RL LQ control regulators, leveraging Hewer’s findings and Theorem 4.

• Remark 4. The issue of complexity in prescribed performance is discussed in. ⁴⁰ The application of adaptive dynamic programing for sliding-mode systems is presented in. ⁴¹

Illustrative example

The selected illustrative example is a dynamic system involving the translational movement of two elastically coupled masses shown in Figure 11. The system consists of two rigid bodies with masses m₁ and m₂, which are connected by a spring with an elasticity coefficient k. The bodies move without friction along a fixed horizontal surface. A force, or control signal u, acts on the left body. This subsystem is very common in various mechatronic systems.

Figure 11.

A mechatronic subsystem as an illustrative example of a controlled plant P _c .

Let x₁ and v₁ be the position and velocity of the left rigid body, and x₂ and v₂ be the position and velocity of the right body. The state vector of the dynamic system is $x={\left[x_1{x}_2{v}_1{v}_2\right]}^T$ , and the continuous model of the dynamic system is described by the state-space model.

$$\begin{array}{cc}\multicolumn{1}{c}{\stackrel{·}{x}=\mathrm{Ax}+\mathrm{Bu},}& A=\left[\begin{array}{cc}\begin{array}{cc}0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\hfill & \begin{array}{cc}1\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 1\hfill \end{array}\hfill \\ \multicolumn{1}{c}{\begin{array}{cc}-\frac{k}{m_1}\hfill & \frac{k}{m_1}\hfill \\ \multicolumn{1}{c}{\frac{k}{m_2}}& -\frac{k}{m_2}\hfill \end{array}}& \begin{array}{cc}0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\hfill \end{array}\right],\hfill \\ B=\left[\begin{array}{c}\begin{array}{c}0\hfill \\ \multicolumn{1}{c}{0}\\ \multicolumn{1}{c}{\frac{1}{m_1}}\end{array}\hfill \\ \multicolumn{1}{c}{0}\end{array}\right]\hfill \end{array}$$

(46)

In this illustrative example, we will adopt the following parameter values $k_1=k_2=m_1=m_2=1$ , which will not reduce the generality of our explanations.

Sampling (46) with a sampling time h and using a FOH gives the discrete-time model:

$$\left[\begin{array}{c}x((k+1)h)\hfill \\ \multicolumn{1}{c}{u(\mathrm{kh})}\end{array}\right]=\left[\begin{array}{cc}A\hfill & B_1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill \end{array}\right]\left[\begin{array}{c}x(\mathrm{kh})\hfill \\ \multicolumn{1}{c}{u((k-1)h)}\end{array}\right]+\left[\begin{array}{c}{B}_2\hfill \\ \multicolumn{1}{c}{I}\end{array}\right]u(\mathrm{kh})$$

where:

$$A=e^{A_{c}h}$$

$$B_1=\underset{0}{\overset{h}{\int }}(\frac{\eta }{h}-1)e^{A_c\eta }{B}_c\mathrm{d}\eta$$

$$B_2=\underset{0}{\overset{h}{\int }}(2-\frac{\eta }{h})e^{A_c\eta }{B}_c\mathrm{d}\eta$$

For λ = 0.30, h = 1, $\epsilon =0.0005$ , P ₀  = I, Q = [1 0 0 0 0; 0 1 0 0 0; 0 0 10 0 0; 0 0 0 10 0; 0 0 0 0 10], R = 0.8, K ₀  = [0.397 0.82 0.27 0.63 0.29], the Algorithm 2 gives K = [0.98671 −0.054444 1.1925 0.89862 −0.26683], in

$$u\left(\mathrm{kh}\right)=-K\left[\begin{array}{c}x\left(\mathrm{kh}\right)\hfill \\ \multicolumn{1}{c}{u\left((k-1)h\right)}\end{array}\right]$$

(47)

That is,

$$\begin{array}{c}\multicolumn{1}{c}{u\left(\mathrm{kh}\right)=0.99x_1\left(\mathrm{kh}\right)-0.05x_2\left(\mathrm{kh}\right)+1.19x_3\left(\mathrm{kh}\right)}\\ +0.9x_4\left(\mathrm{kh}\right)-0.27u(\mathrm{kh}-h)\hfill \end{array}$$

The convergence efficiency of Algorithm 2 is illustrated in Figures 12 and 13.

Figure 12.

Convergence of a vector K for different λ values.

Figure 13.

Motion of the autonomous dynamic system (47 and 48), for initial conditions (x₁, x₂, x₃, x₄) = (−1, 1, 1, −1) and u = 0: Less λ (0.30) provides a faster transient process, whereby the control signal has larger amplitude changes.

From the above figures it is possible to see that for smaller degree of stability the convergence of state of dynamic system is faster.

Conclusions

This paper discusses the design of a reinforcement learning LQ regulator with a focus on two key aspects:

• Many real-world systems inherently operate at multiple rates, offering benefits such as improved stability margins, simultaneous stabilization, ⁴² and decentralized control ⁴³

• The regulator aims to minimize quadratic losses while ensuring that closed-loop poles reside within the specified region of the z-plane (0, 1]. This approach exhibits lower sensitivity to uncertainties in plant parameters compared to conventional methods, although the gain margin may vary.

The study employs a sampled-data model using FOH and introduces an LQ reinforcement learning regulator. It also addresses robustness using lifting techniques. Future research could focus on developing regulator designs that do not require knowledge of the system model, and explore aspects related to the frequency domain. ⁴⁴ Also, interesting directions are the design RL LQ regulators for multivariable and continuous systems.