Hierarchical approximate optimal coordinated control of modular robot manipulators through Nash-Stackelberg differential game

Abstract

The hierarchical approximate optimal coordinated control of modular robot manipulators (MRMs) modeled by joint torque feedback is developed via adaptive dynamic programming (ADP). Taking advantage of the Nash-Stackelberg differential game framework, the coordinated control issue is transformed into Stackelberg differential game between MRM as leader formulated as Nash differential game and coordinate operate objective as follower. The critic neural network (NN) is built to deal with the nonlinear cost function and then derived for obtaining the approximate optimal control strategy. Theoretical analysis and experimental verification have respectively proved the effectiveness of the proposed control algorithm.

Introduction

Modular robot manipulator (MRM)^1,2 refers to a category of robots equipped with standard modules and interfaces, enabling them to recombine and reconfigure their own configurations in accordance with diverse task requirements. Guided by the design concepts of modularity and reconfigurability, each joint module of MRM integrates units for communication, sensing, actuation, and control. Through the reconfiguration of these modules, the robot can adopt multiple assembly configurations to accomplish various work tasks, thereby demonstrating advantages that traditional robots do not possess. Furthermore, robot coordinated operation embodies synergy between the precision and efficiency of robotic system as well as the flexibility and intelligence inherent to human operator.^3,4 Consequently, the collaborative task execution between humans and MRMs with operation object has been continuously advancing and widely applied in various daily application fields such as medical care, industry, and elderly care.

During robot coordinated operation task, both humans and robots make corresponding decisions according to every other’s strategies. Therefore, game theory has employed in recent years to describe the coordinated operation process and quantify interaction behavior that are otherwise difficult to characterize. ⁵ The well-known Nash equilibrium for zero-sum^6,7/non-zero-sum^8,9 games and Pareto equilibrium for cooperative games^10,11 are both achieved through the adoption of differences in the strategic behavior exhibited by participants. A pivotal aspect of the Nash/Pareto equilibrium framework is that, each participant can obtain information fairly within the dynamic system. However, in robot coordinated operation task, humans and robots cannot possess absolutely equal dominant relationship, nor can they perceive all information about each other. ¹² Instead, the follower can only respond to the leader’s actions. Consequently, the aforementioned game approaches, which assume equal roles among participants, are not applicable in this context.

The theoretical framework of the Stackelberg game originates from the 1934 work of German economist Stackelberg and has been applied to address a wide range of economic issues.^13–15 Subsequently, it has also found beyond its core applications to fields like nonlinear process, ¹⁶ biology, ¹⁷ and decision-making issue. ¹⁸ Stackelberg game comprises two distinct types of agents: leaders and followers. Its core characteristic lies in addressing sequential decision-making processes, where the leader formulates and implements its strategy prior to the follower. Following this, the follower attempts to make an optimized response according to leader’s strategy. Simultaneously, the leader adjusts its strategy to generate the optimal response based on follower’s potential actions, according to the perceived information of the follower’s strategy. When humans and MRMs perform coordinated operation tasks, humans can assist robots in operations, thus, humans/coordinate operate objectives are regarded as followers while robots act as leaders. Addressing the problem of finding the optimal control strategies for the coordinated operation system, a crucial step in the analysis involves finding the Stackelberg equilibrium solution within the dynamic system. However, for complex problems such as coordinated operation, the occurrence of the curse of dimensionality means that deriving the optimal solution through analytical equations is not always feasible.^19,20 Therefore, researchers have adopted learning methods based on adaptive dynamic programming (ADP)^21–23 to ensure the system’s convergence to an approximate equilibrium state.

Building upon this foundation, a hierarchical approximate optimal coordinated control strategy is proposed for MRMs modeled with joint torque feedback, according to ADP. Leveraging the Nash-Stackelberg differential game framework, the coordinated control problem is converted into a Stackelberg differential game, where the MRM formulated as Nash differential game acts as the leader and the coordinated operation objective serves as the follower. To handle the nonlinear cost function, a critic NN is constructed; subsequently, the approximate optimal control strategy is obtained with the aid of this critic NN. The performance of the proposed control algorithm is rigorously assessed through theoretical analysis and experimental verification.

The highlights of this article are reflected in following:

The creative Nash-Stackelberg differential game framework is developed for the coordinated operation task, that is with Stackelberg game between MRM acted as Nash game deemed as leader and operation objective as follower.

The developed method with Stackelberg differential game highlights the sequential decision-making process. Besides, the proposed methods have been respectively confirmed by theory and experiments, unlike only in simulation.^24,25

MRM’s dynamic model analysis

Formulation of the MRM dynamics

Following the framework in Ref., ²⁶ we employ the joint torque feedback technique to describe the $i$ th MRM subsystem with the following dynamic model:

$$\begin{array}{c}\multicolumn{1}{c}{I_{\mathrm{imo}}{\gamma }_{\mathrm{ira}}{\stackrel{··}{q}}_i+\frac{{\tau }_{\mathrm{ico}}}{{\gamma }_{\mathrm{ira}}}+f_{\mathrm{ihu}}(q_i,{\stackrel{·}{q}}_i)+I_{\mathrm{idc}}(q,\stackrel{·}{q},\stackrel{··}{q})}\\ \multicolumn{1}{c}{={\tau }_i+{\left[J_{\mathrm{iac}}^T{f}_{\mathrm{co}}\right]}_i,}\end{array}$$

(1)

with the subscript $i$ specifies the respective joint subsystem, $I_{\mathrm{imo}}$ is motor’s moment of inertia, ${\gamma }_{\mathrm{ira}}$ denotes gear ratio, $q_i$ is joint position, ${\tau }_{\mathrm{ico}}$ is coupled joint torque, $f_{\mathrm{ihu}}(q_i,{\stackrel{·}{q}}_i)$ means joint friction, $I_{\mathrm{idc}}(q,\stackrel{·}{q},\stackrel{··}{q})$ represents interconnected dynamic coupling (IDC) effect, ${\tau }_i$ is control torque, $J_{\mathrm{iac}}$ represents Jacobi matrix, $f_{\mathrm{co}}$ represents the interaction force between the MRM and object of coordinated operation, and ${[.]}_i$ is $i$ th element of the vector.

• Assumption 1: The rotor is assumed to be symmetric with respect to its axis of rotation. Joint flexibility is treated as negligible, and torque transmission through the speed reducer is considered to be failure-free. Moreover, the inertia between the torque sensor and the speed reducer is deemed insignificant.

• 1) Joint friction

$f_{\mathrm{ihu}}(q_i,{\stackrel{·}{q}}_i)$ takes the form of:

$$\begin{array}{c}\multicolumn{1}{c}{f_{\mathrm{ihu}}(q_i,{\stackrel{·}{q}}_i)={\hat{f}}_{\mathrm{ibv}}{\stackrel{·}{q}}_i+({\hat{f}}_{\mathrm{ist}}{e}^{(-{\hat{f}}_{i\tau s}{\stackrel{·}{q}}_i^2)}+{\hat{f}}_{\mathrm{ico}})\mathrm{sgn}\left({\stackrel{·}{q}}_i\right)}\\ \multicolumn{1}{c}{+f_{\mathrm{ipd}}(q_i,{\stackrel{·}{q}}_i)+K\left({\stackrel{·}{q}}_i\right){\tilde{F}}_{\mathrm{ipu}},}\end{array}$$

(2)

in which

$$K\left({\stackrel{·}{q}}_i\right)={[f_{\mathrm{ibv}}-{\hat{f}}_{\mathrm{ibv}},f_{\mathrm{ico}}-{\hat{f}}_{\mathrm{ico}},f_{\mathrm{ist}}-{\hat{f}}_{\mathrm{ist}},f_{i\tau s}-{\hat{f}}_{i\tau s}]}^T,$$

(3)

where $f_{\mathrm{ipd}}(q_i,{\stackrel{·}{q}}_i)$ means position dependency friction, $f_{\mathrm{ibv}},f_{i\tau s}$ are viscous and Stribect friction, $f_{\mathrm{ist}},f_{\mathrm{ico}}$ encompass both static and Coulomb friction effects. ${\hat{f}}_{\mathrm{ibv}},{\hat{f}}_{\mathrm{ico}},{\hat{f}}_{\mathrm{ist}},{\hat{f}}_{i\tau s}$ are the estimated values of $f_{\mathrm{ibv}},f_{\mathrm{ico}},f_{\mathrm{ist}},f_{i\tau s}$ .

• Remark 1. $f_{\mathrm{ibv}},f_{\mathrm{ico}},f_{\mathrm{ist}},f_{i\tau s}$ and estimations remain bounded, ${\tilde{F}}_{\mathrm{ipu}}$ is with $\left|{\tilde{F}}_{\mathrm{ipu}}\right|\le b_{\mathrm{ipu}}$ , and $b_{\mathrm{ipu}}$ is known constant. This formulation directly leads to the expression for $K\left({\stackrel{·}{q}}_i\right){\tilde{F}}_{\mathrm{ipu}}$ : $\left|K\left({\stackrel{·}{q}}_i\right){\tilde{F}}_{\mathrm{ipu}}\right|\le K\left({\stackrel{·}{q}}_i\right)b_{\mathrm{ipu}}$ . Besides, $\left|f_{\mathrm{ipd}}(q_i,{\stackrel{·}{q}}_i)\right|\le b_{\mathrm{ipd}}$ , in which $b_{\mathrm{ipd}}$ is constant bound.

• 2) IDC effect

The IDC coupling among joint modules is mathematically characterized by:

$$\begin{array}{c}\multicolumn{1}{c}{I_{\mathrm{idc}}=I_{\mathrm{imo}}\sum _{j=1}^{i-1}{v}_{\mathrm{mi}}^T{v}_{\mathrm{lj}}{\stackrel{··}{q}}_j+I_{\mathrm{imo}}\sum _{j=2}^{i-1}\sum _{k=1}^{j-1}{v}_{\mathrm{mi}}^T(v_{\mathrm{lk}}\times v_{\mathrm{lj}}){\stackrel{·}{q}}_k{\stackrel{·}{q}}_j}\\ \multicolumn{1}{c}{=I_{\mathrm{imo}}\sum _{j=1}^{i-1}{D}_j^i{\stackrel{··}{q}}_j+I_{\mathrm{imo}}\sum _{j=2}^{i-1}\sum _{k=1}^{j-1}{\mathrm{\Theta }}_{\mathrm{kj}}^i{\stackrel{·}{q}}_k{\stackrel{·}{q}}_j}\\ \multicolumn{1}{c}{=\sum _{j=1}^{i-1}[I_{\mathrm{imo}}{\hat{D}}_j^i,I_{\mathrm{imo}}]{[{\stackrel{··}{q}}_j,{\tilde{D}}_j^i{\stackrel{··}{q}}_j]}^T}\\ \multicolumn{1}{c}{+\sum _{j=2}^{i-1}\sum _{k=1}^{j-1}[I_{\mathrm{imo}}{\hat{\mathrm{\Theta }}}_{\mathrm{kj}}^i,I_{\mathrm{imo}}]{[{\stackrel{··}{q}}_j,{\tilde{\mathrm{\Theta }}}_{\mathrm{kj}}^i{\stackrel{·}{q}}_k{\stackrel{·}{q}}_j]}^T,}\end{array}$$

(4)

in which $v_{\mathrm{mi}},v_{\mathrm{lj}},v_{\mathrm{lk}}$ are denote the unit vectors along the rotational axes of the $i$ th motor, the $j$ th link, and the $k$ th link. Then, we have $D_j^i=v_{\mathrm{mi}}^T{v}_{\mathrm{lj}}$ as well as ${\mathrm{\Theta }}_{\mathrm{kj}}^i=v_{\mathrm{mi}}^T(v_{\mathrm{lk}}\times v_{\mathrm{lj}})$ . Additionally, the following relations hold: that ${\hat{D}}_j^i=D_j^i-{\tilde{D}}_j^i$ and ${\hat{\mathrm{\Theta }}}_{\mathrm{kj}}^i={\mathrm{\Theta }}_{\mathrm{kj}}^i-{\tilde{\mathrm{\Theta }}}_{\mathrm{kj}}^i$ , where ${\hat{D}}_j^i,{\hat{\mathrm{\Theta }}}_{\mathrm{kj}}^i$ denote estimated values of $D_j^i,{\mathrm{\Theta }}_{\mathrm{kj}}^i$ and ${\tilde{D}}_j^i,{\tilde{\mathrm{\Theta }}}_{\mathrm{kj}}^i$ represent alignment errors.

• Remark 2. From the definitions $\left|D_j^i\right|=\left|v_{\mathrm{mi}}^T{v}_{\mathrm{lj}}\right|<1$ and $\left|{\mathrm{\Theta }}_{\mathrm{kj}}^i\right|=\left|v_{\mathrm{mi}}^T(v_{\mathrm{lk}}\times v_{\mathrm{lj}})\right|<1$ based on vectors $v_{\mathrm{mi}},v_{\mathrm{lk}},v_{\mathrm{lj}}$ , we derive the boundedness of $I_{\mathrm{idc}}(q,\stackrel{·}{q},\stackrel{··}{q})$ , specifically $\left|I_{\mathrm{idc}}(q,\stackrel{·}{q},\stackrel{··}{q})\right|\le b_{\mathrm{idc}}$ for a bound $b_{\mathrm{idc}}$ .

Define state vector $x_i=[x_{i1},x_{i2}{]}^T=[q_i,{\stackrel{·}{q}}_i{]}^T$ and control input $u_i={\tau }_i$ , $c_i={\left[J_{\mathrm{iac}}^T{f}_{\mathrm{co}}\right]}_i$ . The dynamics of the $i$ th subsystem are characterized by the state space:

$$\begin{array}{c}\multicolumn{1}{c}{\{\begin{array}{c}{\stackrel{·}{x}}_{i1}=x_{i2}\hfill \\ \multicolumn{1}{c}{{\stackrel{·}{x}}_{i2}={\ell }_i\left(x\right)+g_i{u}_i+c_i}\end{array},}\end{array}$$

(5)

where

$$\begin{array}{c}\multicolumn{1}{c}{g_i={\left(I_{\mathrm{imo}}{\gamma }_{\mathrm{ira}}\right)}^{-1}}\\ {\ell }_i=-g_i\left(\begin{array}{c}({\hat{f}}_{\mathrm{ist}}{e}^{(-{\hat{f}}_{i\tau s}{\stackrel{·}{x}}_{i1}^2)}+{\hat{f}}_{\mathrm{ico}})\mathrm{sgn}\left(x_{i2}\right)+f_{\mathrm{ipd}}(x_{i1},x_{i2})\hfill \\ \multicolumn{1}{c}{+{\hat{f}}_{\mathrm{ibv}}{x}_{i2}+K\left(x_{i2}\right){\tilde{F}}_{\mathrm{ipu}}+\frac{{\tau }_{\mathrm{ico}}}{{\gamma }_{\mathrm{ira}}}+I_{\mathrm{idc}}(x,\stackrel{·}{x},\stackrel{··}{x})}\end{array}\right).\hfill \end{array}$$

(6)

In this paper, the primary control objective of this work is to optimally guarantee that the tracking error of MRM system in coordinated operation task is UUB. To address this challenge, the next section proposes a hierarchical approximate optimal coordinated control approach based on Nash-Stackelberg differential game.

Nash-Stackelberg differential game-based hierarchical approximate optimal coordinated control approach of MRM system

Derivation of Nash-Stackelberg differential game

This investigation focuses on leveraging Nash-Stackelberg differential game for approximating optimal coordinated control that MRM can optimally ensure systems’ tracking error UUB. Furthermore, Nash-Stackelberg differential game features a leader-follower hierarchy. The leader prioritizes strategy selection. The follower subsequently determines its optimal strategy in reaction to the leader’s decision. The leader, in turn, formulates its best response by anticipating the follower’s strategic behavior. In the MRM’s coordinated operation task, MRM is deemed as the leader, and the coordinated operation object is regarded as the follower. Besides, MRM system is composed of $n$ modules, which are players in the Nash differential game to optimize the global leader’s performance. Therefore, a joint Nash-Stackelberg differential game is developed in this section.

The diagram of the Nash-Stackelberg differential game-based hierarchical approximate optimal coordinated control is illustrated in Figure 1. Each joint of the MRM system (the leader) is regarded as a player in Nash differential game, while forming Stackelberg game with the object being manipulated (the follower). Consider a scenario where a robot and a human are required to transport a large object that neither could handle independently. In this setup, a desired trajectory is prescribed for the robot, and the manipulated object is designed to follow the robot’s position, thereby establishing hierarchical control framework (Stackelberg game). By employing ADP algorithm, critic neural networks are utilized to approximate the Hamiltonian functions of both the leader and the follower. The resulting residuals are then used to adjust the update laws of the critic networks, ultimately yielding approximate optimal control laws for both the leader and the follower.

Figure 1.

The diagram of the Nash-Stackelberg differential game-based hierarchical approximate optimal coordinated control.

According to (5), one expresses the overall state space equation is as follow:

$$\begin{array}{c}\multicolumn{1}{c}{\{\begin{array}{c}{\stackrel{·}{x}}_1=x_2\hfill \\ \multicolumn{1}{c}{{\stackrel{·}{x}}_2=L\left(x\right)+\sum _{v=1}^n{G}_v{u}_v+C\left(x\right)}\end{array},}\end{array}$$

(7)

where $x=[x_1^T,x_2^T{]}^T\in R^{2n}$ is the global state, $x_1,x_2$ represent $x_1=[x_{11},\dots ,x_{i1},...,x_{n1}{]}^T\in R^n$ and $x_2=[x_{12},\dots ,x_{i2},\dots ,x_{n2}{]}^T\in R^n$ . Then $L\left(x\right)=[{\ell }_1\left(x\right),\dots ,{\ell }_i\left(x\right),\dots ,{\ell }_n\left(x\right){]}^T$ , $G_v=[0,\dots ,0,g_v,0,{...,0]}^T$ , $C\left(x\right)=[c_1,\dots ,c_i,\dots ,c_n{]}^T$ , where $g_v={\left(I_{\mathrm{vmo}}{\gamma }_{\mathrm{vra}}\right)}^{-1},v=1,\dots ,n$ .

Define the cost function of coordinated operation object and MRM system as:

$$\begin{array}{c}\multicolumn{1}{c}{V_{\mathrm{iF}}(\stackrel{·}{E},u_i,C)={\int }_t^{\infty }{r}_{\mathrm{iF}}(\stackrel{·}{E},u_i,C)d\tau }\\ \multicolumn{1}{c}{={\int }_t^{\infty }({\stackrel{·}{E}}^T{Q}_{\mathrm{iF}}\stackrel{·}{E}+\sum _{v=1}^n{u}_{\mathrm{iv}}^T{R}_{\mathrm{iFv}}{u}_{\mathrm{iv}}+C^T{R}_{\mathrm{iF}}C)d\tau ,}\end{array}$$

(8)

$$\begin{array}{c}\multicolumn{1}{c}{V_{\mathrm{iL}}(\stackrel{·}{E},u_i,C)={\int }_t^{\infty }{r}_{\mathrm{iL}}(\stackrel{·}{E},u_i,C)d\tau }\\ \multicolumn{1}{c}{={\int }_t^{\infty }({\stackrel{·}{E}}^T{Q}_{\mathrm{iL}}\stackrel{·}{E}+\sum _{v=1}^n{u}_{\mathrm{iv}}^T{R}_{\mathrm{iLv}}{u}_{\mathrm{iv}}+C^T{R}_{\mathrm{iL}}C)d\tau ,}\end{array}$$

(9)

where $r_{\mathrm{iF}}$ and $r_{\mathrm{iL}}$ correspond to the utility functions of the follower and the leader. $e_i=x_{i1}-x_{\mathrm{id}}$ , ${\stackrel{·}{e}}_i=x_{i2}-{\stackrel{·}{x}}_{\mathrm{id}}$ mean position as well as velocity error, $x_{\mathrm{id}}$ represents the desired position in coordinated operation. $Q_{\mathrm{iF}},Q_{\mathrm{iL}},R_{\mathrm{iFv}},R_{\mathrm{iLv}},R_{\mathrm{iF}},R_{\mathrm{iL}}$ are determined positive definite matrices. $E=[e_1,e_2,\cdots ,e_n{]}^T$ .

The coordinated operation object is seeking an optimal control policy for the MRM in response to the current system state, formulated as the problem of minimizing a cost function defined by:

$$V_{\mathrm{iF}}^{*}(\stackrel{·}{E},u_i,C)=\underset{C}{\min }{\int }_t^{\infty }{r}_{\mathrm{iF}}(\stackrel{·}{E},u_i,C)d\tau .$$

(10)

The follower’s Hamiltonian function is constructed by combining the infinitesimal version of (10) with equations (1) and (5):

$$\begin{array}{c}\multicolumn{1}{c}{H_{\mathrm{iF}}(\stackrel{·}{E},u_i,C,\nabla V_{\mathrm{iF}})=r_{\mathrm{iF}}(\stackrel{·}{E},u_i,C)}\\ \multicolumn{1}{c}{+\nabla V_{\mathrm{iF}}(L\left(x\right)+\sum _{v=1}^n{G}_v{u}_v+C\left(x\right)-{\stackrel{··}{x}}_d).}\end{array}$$

(11)

According to the stationary condition $\frac{\partial H_{\mathrm{iF}}}{\partial c_i}=0$ , local follower’s optimal control policy can be determined by applying the minimum principle:

$$c_i^{*}=-\frac{1}{2}{R}_{\mathrm{iF}}^{-1}\nabla V_{\mathrm{iF}}^{*}.$$

(12)

Subsequently, a costate $\hslash$ is defined to incorporate the follower’s future effects, enabling the leader to respond strategically in its decision-making. It guarantees MRM’s requirement for coordinate operation objective’s cost function.

$$\begin{array}{c}\multicolumn{1}{c}{{\stackrel{·}{\hslash }}_{i1}=-\frac{\partial H_{\mathrm{iF}}}{\partial {\stackrel{·}{e}}_i}}\\ \multicolumn{1}{c}{=-\nabla {\ell }_i^T\nabla V_{\mathrm{iF}}^{*}-2Q_{\mathrm{iF}}{\stackrel{·}{e}}_i-g_i^T\nabla V_{\mathrm{iF}}^{*}{u}_i-\nabla V_{\mathrm{iF}}^{*}{c}_i.}\end{array}$$

(13)

Based on costate (13) and ${\stackrel{·}{x}}_2=L\left(x\right)+\underset{v=1}{\sum ^n}{G}_v{u}_v+C^{*}\left(x\right)$ , this allows for the derivation of the leader’s optimal cost function:

$$V_{\mathrm{iL}}^{*}(\stackrel{·}{E},u_i,C)=\underset{u_i}{\min }{\int }_t^{\infty }(r_{\mathrm{iL}}(\stackrel{·}{E},u_i,C)+{\lambda }_i^T{\stackrel{·}{\hslash }}_{i1})d\tau ,$$

(14)

where ${\lambda }_i$ is Lagrange multiplier regarding to (13).

By an analogous procedure, the leader’s Hamiltonian function and optimal control policy can be formulated:

$$\begin{array}{c}\multicolumn{1}{c}{H_{\mathrm{iL}}(\stackrel{·}{E},u_i,C,\nabla V_{\mathrm{iL}})=r_{\mathrm{iL}}(\stackrel{·}{E},u_i,C)}\\ \multicolumn{1}{c}{+\nabla V_{\mathrm{iL}}^T(L\left(x\right)+\sum _{v=1}^n{G}_v{u}_v+C\left(x\right)-{\stackrel{··}{x}}_d)}\\ \multicolumn{1}{c}{+{\lambda }_i^T{\stackrel{·}{\hslash }}_{i1},}\end{array}$$

(15)

$$\frac{\partial H_{\mathrm{iL}}}{\partial u_i}=0\to u_i^{*}=-\frac{1}{2}{R}_{\mathrm{iLi}}^{-1}{g}_i^T(\nabla V_{\mathrm{iL}}^{*}-\nabla V_{\mathrm{iF}}^{*}{\lambda }_i).$$

(16)

The behavior of the costate is characterized by:

$${\stackrel{·}{\hslash }}_{i2}=-{\left(\frac{\partial H_{\mathrm{iL}}}{\partial {\stackrel{·}{e}}_i}\right)}^T=-\nabla {\ell }_i^T\nabla V_{\mathrm{iL}}^{*}-2Q_{\mathrm{iL}}{\stackrel{·}{e}}_i+\frac{\partial {\stackrel{·}{\hslash }}_{i1}}{\partial {\stackrel{·}{e}}_i}{\lambda }_i,$$

(17)

$$\begin{array}{c}\multicolumn{1}{c}{{\stackrel{·}{\lambda }}_i=-{\left(\frac{\partial H_{\mathrm{iL}}}{\partial V_{\mathrm{iF}}^{*}}\right)}^T=-\frac{1}{2}{g}_i\left(x_i\right)R_{\mathrm{iF}}^{-1}{R}_{\mathrm{iLi}}{R}_{\mathrm{iF}}^{-1}{g}_i^T\nabla V_{\mathrm{iF}}^{*}}\\ \multicolumn{1}{c}{+\frac{1}{2}{g}_i\left(x_i\right)R_{\mathrm{iF}}^{-1}{g}_i^T\left(x_i\right)\nabla V_{\mathrm{iL}}^{*}+\nabla {\ell }_i{\lambda }_i+g_i{u}_i{\lambda }_i+c_i{\lambda }_i.}\end{array}$$

(18)

The coupled Hamilton-Jacobi equations are obtained by substituting policies (12) and (16) into their respective Hamiltonian functions (11) and (15), leading to:

$$\begin{array}{c}\multicolumn{1}{c}{0=r_{\mathrm{iF}}(\stackrel{·}{E},u_i^{*},C_{*})}\\ \multicolumn{1}{c}{+\nabla V_{\mathrm{iF}}(L\left(x\right)+\sum _{v=1}^n{G}_v{u}_v+C\left(x\right)-{\stackrel{··}{x}}_d),}\end{array}$$

(19)

$$\begin{array}{c}\multicolumn{1}{c}{0=r_{\mathrm{iL}}(\stackrel{·}{E},u_i^{*},C_{*})}\\ \multicolumn{1}{c}{+\nabla V_{\mathrm{iL}}(L\left(x\right)+\sum _{v=1}^n{G}_v{u}_v+C\left(x\right)-{\stackrel{·}{x}}_d)}\\ \multicolumn{1}{c}{+{\lambda }_i^T{\stackrel{·}{\hslash }}_{i1}.}\end{array}$$

(20)

The solution to the coupled Hamilton-Jacobi equations (19) and (20) for $\nabla V_{\mathrm{iF}}$ and $\nabla V_{\mathrm{iL}}$ provides the Nash-Stackelberg equilibrium, and leads to an approximate optimal control that guarantees MRM tracking performance in coordinated operations. To circumvent the challenge of solving these complex equations directly, the critic neural network approximation approach is adopted.

Approximate solution of the Nash-Stackelberg differential game-based hierarchical approximate optimal coordinated control via the implementation of critic NN

From decomposition (16), one obtains the representation:

$$u_i^{*}=u_{i1}+u_{i2}^{*},$$

(21)

where $u_{i1}$ is handling with ${\ell }_i\left(x\right)$ , and $u_{i2}^{*}$ utilizes optimal compensation of coordinated operation task.

According to (6), $u_{i1}$ can be designed as:

$$u_{i1}=-\left(\begin{array}{c}-({\hat{f}}_{\mathrm{ist}}{e}^{(-{\hat{f}}_{i\tau s}{x}_{i2}^2)}+{\hat{f}}_{\mathrm{ico}})\mathrm{sgn}\left(x_{i2}\right)\hfill \\ \multicolumn{1}{c}{-{\hat{f}}_{\mathrm{ibv}}{x}_{i2}-g_i^{-1}{\stackrel{··}{x}}_{\mathrm{id}}-\frac{{\tau }_{\mathrm{ico}}}{{\gamma }_{\mathrm{ira}}}}\end{array}\right).$$

(22)

The optimal compensation control issue is thereby recasted as a hierarchical approximate optimal control problem based on Nash-Stackelberg differential game. To address this, critic neural network is employed to approximate the corresponding cost functions (10) and (14):

$$V_{\mathrm{im}}^{*}\left(\stackrel{·}{E}\right)=W_{\mathrm{imcr}}^T{\varphi }_{\mathrm{imcr}}\left(\stackrel{·}{E}\right)+{\epsilon }_{\mathrm{imcr}},m=F,L,$$

(23)

where $W_{\mathrm{imcr}}$ is denoted as the critic NN vector, ${\epsilon }_{\mathrm{imcr}}$ represents approximate error, ${\varphi }_{\mathrm{imcr}}\left(\stackrel{·}{E}\right)$ is activation function.

The gradient of the expression in (23) is obtained as follows:

$$\nabla V_{\mathrm{im}}^{*}\left(\stackrel{·}{E}\right)=\nabla {\varphi }_{\mathrm{imcr}}\left(\stackrel{·}{E}\right)W_{\mathrm{imcr}}+\nabla {\epsilon }_{\mathrm{imcr}},m=F,L,$$

(24)

where $\nabla {\varphi }_{\mathrm{imcr}}\left(\stackrel{·}{E}\right)=\partial {\varphi }_{\mathrm{imcr}}\left(\stackrel{·}{E}\right)/\partial \stackrel{·}{E}$ is gradient of activation function, $\nabla {\epsilon }_{\mathrm{imcr}}$ is gradient error.

By substituting (24) into (12) and (16), the optimal control torques are given by:

$$c_i^{*}=-\frac{1}{2}{R}_{\mathrm{iF}}^{-1}(\nabla {\varphi }_{\mathrm{iFcr}}\left(\stackrel{·}{E}\right)W_{\mathrm{iFcr}}+\nabla {\epsilon }_{\mathrm{iFcr}}),$$

(25)

$$\begin{array}{c}\multicolumn{1}{c}{u_{i2}^{*}=-\frac{1}{2}{R}_{\mathrm{iLi}}^{-1}{g}_i^T(\nabla {\varphi }_{\mathrm{iLcr}}\left(\stackrel{·}{E}\right)W_{\mathrm{iLcr}}+\nabla {\epsilon }_{\mathrm{iLcr}})}\\ \multicolumn{1}{c}{+\frac{1}{2}{R}_{\mathrm{iLi}}^{-1}{g}_i^T(\nabla {\varphi }_{\mathrm{iFcr}}\left(\stackrel{·}{E}\right)W_{\mathrm{iFcr}}+\nabla {\epsilon }_{\mathrm{iFcr}}){\lambda }_i.}\end{array}$$

(26)

Substituting (25), (26) into (11) and (15), one yields:

$$\begin{array}{c}\multicolumn{1}{c}{H_{\mathrm{iF}}(\stackrel{·}{E},u_i,C,W_{\mathrm{iFcr}})=r_{\mathrm{iF}}(\stackrel{·}{E},u_i,C)+\nabla {\varphi }_{\mathrm{iFcr}}\left(\stackrel{·}{E}\right)}\\ \multicolumn{1}{c}{W_{\mathrm{iFcr}}(L\left(x\right)+\sum _{v=1}^n{G}_v{u}_v+C\left(x\right)-{\stackrel{··}{x}}_d)=e_{\mathrm{iHF}},}\end{array}$$

(27)

$$\begin{array}{c}\multicolumn{1}{c}{H_{\mathrm{iL}}(\stackrel{·}{E},u_i,C,W_{\mathrm{iLcr}})=r_{\mathrm{iL}}(\stackrel{·}{E},u_i,C)+\nabla {\varphi }_{\mathrm{iLcr}}\left(\stackrel{·}{E}\right)W_{\mathrm{iLcr}}}\\ \multicolumn{1}{c}{(L\left(x\right)+C(x)-{\stackrel{··}{x}}_d+\sum _{v=1}^n{G}_v{u}_v)}\\ \multicolumn{1}{c}{+(-\nabla {\ell }_i^T\nabla {\varphi }_{\mathrm{iFcr}}\left(\stackrel{·}{E}\right)W_{\mathrm{iFcr}}}\\ \multicolumn{1}{c}{-2Q_{\mathrm{iF}}{\stackrel{·}{e}}_i-g_i^T\nabla {\varphi }_{\mathrm{iFcr}}\left(\stackrel{·}{E}\right)W_{\mathrm{iFcr}}{u}_i}\\ \multicolumn{1}{c}{-\nabla {\varphi }_{\mathrm{iFcr}}\left(\stackrel{·}{E}\right)W_{\mathrm{iFcr}}{c}_i)}\\ \multicolumn{1}{c}{{\lambda }_i=e_{\mathrm{iHL}},}\end{array}$$

(28)

where $e_{\mathrm{iHF}}$ and $e_{\mathrm{iHL}}$ are residual errors.

The estimated optimal cost function is:

$${\hat{V}}_{\mathrm{im}}^{*}\left(\stackrel{·}{E}\right)={\hat{W}}_{\mathrm{imcr}}^T{\varphi }_{\mathrm{imcr}}\left(\stackrel{·}{E}\right),m=F,L.$$

(29)

According to (25), (26) as well as (29), approximate optimal control policies are:

$${\hat{c}}_i^{*}=-\frac{1}{2}{R}_{\mathrm{iF}}^{-1}(\nabla {\varphi }_{\mathrm{iFcr}}\left(\stackrel{·}{E}\right){\hat{W}}_{\mathrm{iFcr}}),$$

(30)

$$\begin{array}{c}\multicolumn{1}{c}{{\hat{u}}_{i2}^{*}=-\frac{1}{2}{R}_{\mathrm{iLi}}^{-1}{g}_i^T(\nabla {\varphi }_{\mathrm{iLcr}}\left(\stackrel{·}{E}\right){\hat{W}}_{\mathrm{iLcr}})}\\ \multicolumn{1}{c}{+\frac{1}{2}{R}_{\mathrm{iLi}}^{-1}{g}_i^T(\nabla {\varphi }_{\mathrm{iFcr}}\left(\stackrel{·}{E}\right){\hat{W}}_{\mathrm{iFcr}}){\lambda }_i.}\end{array}$$

(31)

Applying (30) and (31) into (27) and (28), the approximated Hamiltonian functions are given by:

$$\begin{array}{c}\multicolumn{1}{c}{{\hat{H}}_{\mathrm{iF}}(\stackrel{·}{E},{\hat{u}}_i,\hat{C},{\hat{W}}_{\mathrm{iFcr}})=r_{\mathrm{iF}}(\stackrel{·}{E},{\hat{u}}_i,\hat{C})+\nabla {\varphi }_{\mathrm{iFcr}}\left(\stackrel{·}{E}\right)}\\ \multicolumn{1}{c}{{\hat{W}}_{\mathrm{iFcr}}(L\left(x\right)+\sum _{v=1}^n{G}_v{\hat{u}}_v+\hat{C}\left(x\right)-{\stackrel{··}{x}}_d)=e_{\mathrm{iF}},}\end{array}$$

(32)

$$\begin{array}{c}\multicolumn{1}{c}{{\hat{H}}_{\mathrm{iL}}(\stackrel{·}{E},{\hat{u}}_i,\hat{C},{\hat{W}}_{\mathrm{iLcr}})=r_{\mathrm{iL}}(\stackrel{·}{E},{\hat{u}}_i,\hat{C})+\nabla {\varphi }_{\mathrm{iLcr}}\left(\stackrel{·}{E}\right){\hat{W}}_{\mathrm{iLcr}}}\\ \multicolumn{1}{c}{(L\left(x\right)+\hat{C}(x)-{\stackrel{··}{x}}_d+\sum _{v=1}^n{G}_v{\hat{u}}_v)+(-\nabla {\ell }_i^T\nabla {\varphi }_{\mathrm{iFcr}}\left(\stackrel{·}{E}\right){\hat{W}}_{\mathrm{iFcr}}}\\ \multicolumn{1}{c}{-2Q_{\mathrm{iF}}{\stackrel{·}{e}}_i-g_i^T\nabla {\varphi }_{\mathrm{iFcr}}\left(\stackrel{·}{E}\right){\hat{W}}_{\mathrm{iFcr}}{\hat{u}}_i-\nabla {\varphi }_{\mathrm{iFcr}}\left(\stackrel{·}{E}\right){\hat{W}}_{\mathrm{iFcr}}{\hat{c}}_i)}\\ \multicolumn{1}{c}{{\lambda }_i=e_{\mathrm{iL}},}\end{array}$$

(33)

The approximated Hamiltonian error functions $e_{\mathrm{iF}}$ as well as $e_{\mathrm{iL}}$ are defined as follow equation:

$$e_{\mathrm{im}}={\hat{H}}_{\mathrm{im}}-H_{\mathrm{im}},m=F,L,$$

(34)

where $e_{\mathrm{im}}={\hat{H}}_{\mathrm{im}}$ is gotten from (27), (28), (32) and (33).

When defining ${\tilde{W}}_{\mathrm{imcr}}=W_{\mathrm{imcr}}-{\hat{W}}_{\mathrm{imcr}},$ it can be deduced that $e_{\mathrm{im}}=e_{\mathrm{iHm}}-{\tilde{W}}_{\mathrm{imcr}}^T\nabla {\varphi }_{\mathrm{imcr}}\left(\stackrel{·}{E}\right)\stackrel{··}{E}$ , by (27), (28), (32), (33) with (34). According to gradient decent, define residual error function $E_{\mathrm{im}}=\frac{1}{2}{e}_{\mathrm{im}}^2$ . To minimize this cost function, the critic NN weights are adjusted according to the following update law:

$${\stackrel{·}{\hat{W}}}_{\mathrm{imcr}}=-\varsigma e_{\mathrm{im}}\nabla {\varphi }_{\mathrm{imcr}}\left(\stackrel{·}{E}\right)\stackrel{··}{E},$$

(35)

where ${\varsigma }_{\mathrm{imcr}}$ is updated rate of critic NN. ${\xi }_{\mathrm{imcr}}$ represents $\nabla {\varphi }_{\mathrm{imcr}}\left(\stackrel{·}{E}\right)\stackrel{··}{E}$ , and positive constant ${\xi }_{\mathrm{iLm}}$ that $\parallel {\xi }_{\mathrm{imcr}}\parallel \le {\xi }_{\mathrm{iLm}}$ is assumed. Then, we have

$$\begin{array}{c}\multicolumn{1}{c}{{\stackrel{·}{\stackrel{~}{W}}}_{\mathrm{imcr}}=-{\stackrel{·}{\hat{W}}}_{\mathrm{imcr}}={\varsigma }_{\mathrm{imcr}}{e}_{\mathrm{im}}\nabla {\varphi }_{\mathrm{imcr}}\left(\stackrel{·}{E}\right)\stackrel{··}{E}}\\ \multicolumn{1}{c}{={\varsigma }_{\mathrm{imcr}}(e_{\mathrm{iHm}}-{\tilde{W}}_{\mathrm{imcr}}^T{\xi }_{\mathrm{imcr}}){\xi }_{\mathrm{imcr}}.}\end{array}$$

(36)

• Remark 3. The convergence properties of the critic neural network are contingent upon the satisfaction of the persistent excitation condition. This condition, which ensures adequate exploration of the system’s dynamic, is intrinsically ensured within the context of trajectory tracking control task due to the continuous time-varying nature of the reference path.

• Theorem 1. Considered $V_{\mathrm{im}}$ which is approximated by (23), with $W_{\mathrm{imcr}}$ , and ${\hat{V}}_{\mathrm{im}}$ given by (29) built with ${\hat{W}}_{\mathrm{imcr}}$ . Updating the critic NN weight via law (35) ensures UUB of weight approximation error.

Proof. The following Lyapunov function candidate is selected for stability analysis:

$$L_{\mathrm{im}}\left(t\right)=\frac{1}{2{\varsigma }_{\mathrm{imcr}}}{\tilde{W}}_{\mathrm{imcr}}^T{\tilde{W}}_{\mathrm{imcr}},m=F,L.$$

(37)

Time derivative of $L_{\mathrm{im}}\left(t\right)$ has:

$$\begin{array}{c}\multicolumn{1}{c}{{\stackrel{·}{L}}_{\mathrm{im}}\left(t\right)=\frac{1}{{\varsigma }_{\mathrm{imcr}}}{\tilde{W}}_{\mathrm{imcr}}^T{\stackrel{·}{\stackrel{~}{W}}}_{\mathrm{imcr}}}\\ \multicolumn{1}{c}{={\tilde{W}}_{\mathrm{imcr}}^T(e_{\mathrm{iHm}}-{\tilde{W}}_{\mathrm{imcr}}^T{\xi }_{\mathrm{imcr}}){\xi }_{\mathrm{imcr}}}\\ \multicolumn{1}{c}{={\tilde{W}}_{\mathrm{imcr}}^T{e}_{\mathrm{iHm}}{\xi }_{\mathrm{imcr}}-\parallel {\tilde{W}}_{\mathrm{imcr}}^T{\xi }_{\mathrm{imcr}}{\parallel }^2}\\ \multicolumn{1}{c}{\le \frac{1}{2}{e}_{\mathrm{iHm}}^2-\frac{1}{2}\parallel {\tilde{W}}_{\mathrm{imcr}}^T{\xi }_{\mathrm{imcr}}{\parallel }^2,m=F,L,}\end{array}$$

(38)

where ${\stackrel{·}{L}}_{\mathrm{im}}\left(t\right)\le 0$ when $e_{\mathrm{iHm}}$ lies outside ${\mathrm{\Omega }}_{\mathrm{im}}=\{{\tilde{W}}_{\mathrm{imcr}}:\parallel {\tilde{W}}_{\mathrm{imcr}}\parallel \le \frac{e_{\mathrm{iHm}}}{{\xi }_{\mathrm{iLm}}}\}$ . Proof complete.

According to (22) and the hierarchical optimal coordinated control (31), ${\hat{u}}_i^{*}$ is:

$$\begin{array}{c}\multicolumn{1}{c}{{\hat{u}}_i^{*}=u_{i1}+{\hat{u}}_{i2}^{*}}\\ \multicolumn{1}{c}{=-\left(\begin{array}{c}-({\hat{f}}_{\mathrm{ist}}{e}^{(-{\hat{f}}_{i\tau s}{x}_{i2}^2)}+{\hat{f}}_{\mathrm{ico}})\mathrm{sgn}\left(x_{i2}\right)\hfill \\ \multicolumn{1}{c}{-{\hat{f}}_{\mathrm{ibv}}{x}_{i2}-g_i^{-1}{\stackrel{··}{x}}_{\mathrm{id}}-\frac{{\tau }_{\mathrm{ico}}}{{\gamma }_{\mathrm{ira}}}}\end{array}\right)}\\ \multicolumn{1}{c}{-\frac{1}{2}{R}_{\mathrm{iLi}}^{-1}{g}_i^T(\nabla {\varphi }_{\mathrm{iLcr}}\left(\stackrel{·}{E}\right){\hat{W}}_{\mathrm{iLcr}})}\\ \multicolumn{1}{c}{+\frac{1}{2}{R}_{\mathrm{iLi}}^{-1}{g}_i^T(\nabla {\varphi }_{\mathrm{iFcr}}\left(\stackrel{·}{E}\right){\hat{W}}_{\mathrm{iFcr}}){\lambda }_i.}\end{array}$$

(39)

• Theorem 2. Under the hierarchical control policy (39) derived from the Nash-Stackelberg differential game, the closed-loop dynamics of the MRM system (1) achieve position tracking error UUB during coordinated operations.

Proof. The following Lyapunov function candidate is selected for stability analysis:

$$V_{\mathrm{Ly}}=\sum _{i=1}^n\sum _m^{L,F}{V}_{\mathrm{im}}^{*}.$$

(40)

Derivative (40) as follow equation:

$$\begin{array}{c}\multicolumn{1}{c}{{\stackrel{·}{V}}_{\mathrm{iLy}}\left(t\right)=}\\ \multicolumn{1}{c}{\sum _m^{L,F}\left({(\nabla V_{\mathrm{im}}^{*})}^T(L\left(x\right)+\sum _{v=1}^n{G}_v{u}_v+C\left(x\right)-{\stackrel{··}{x}}_d)\right).}\end{array}$$

(41)

Base on HJ equations in (19) as well as (20), one has:

$$\begin{array}{c}\multicolumn{1}{c}{{(\nabla V_{\mathrm{iF}}^{*})}^T({\ell }_i\left(x\right)-{\stackrel{··}{x}}_{\mathrm{id}})=-{\stackrel{·}{E}}^T{Q}_{\mathrm{iF}}\stackrel{·}{E}+\frac{1}{4}{(\nabla V_{\mathrm{iF}}^{*})}^T}\\ \multicolumn{1}{c}{g_i{R}_{\mathrm{iF}}^{-1}{g}_i^T(\nabla V_{\mathrm{iF}}^{*})+\frac{1}{4}{(\nabla V_{\mathrm{iF}}^{*})}^T{c}_i{R}_{\mathrm{iFi}}^{-1}{c}_i^T(\nabla V_{\mathrm{iF}}^{*}).}\end{array}$$

(42)

$$\begin{array}{c}\multicolumn{1}{c}{{(\nabla V_{\mathrm{iL}}^{*})}^T({\ell }_i\left(x\right)-{\stackrel{··}{x}}_{\mathrm{id}})=-{\stackrel{·}{E}}^T{Q}_{\mathrm{iL}}\stackrel{·}{E}+\frac{1}{4}{(\nabla V_{\mathrm{iL}}^{*})}^T{g}_i}\\ \multicolumn{1}{c}{R_{\mathrm{iL}}^{-1}{g}_i^T(\nabla V_{\mathrm{iL}}^{*})+\frac{1}{4}{(\nabla V_{\mathrm{iL}}^{*})}^T{c}_i{R}_{\mathrm{iLi}}^{-1}{c}_i^T(\nabla V_{\mathrm{iL}}^{*})+{\lambda }_i^T{\stackrel{·}{\hslash }}_{i1}.}\end{array}$$

(43)

Combining (42), (43) into (41), we obtain:

$$\begin{array}{c}\multicolumn{1}{c}{{\stackrel{·}{V}}_{\mathrm{iLy}}\left(t\right)=-{\stackrel{·}{E}}^T(Q_{\mathrm{iF}}+Q_{\mathrm{iL}})\stackrel{·}{E}+\frac{1}{4}{(\nabla V_{\mathrm{iF}}^{*})}^T{g}_i{R}_{\mathrm{iF}}^{-1}{g}_i^T}\\ \multicolumn{1}{c}{(\nabla V_{\mathrm{iF}}^{*})+\frac{1}{4}{(\nabla V_{\mathrm{iF}}^{*})}^T{c}_i{R}_{\mathrm{iFi}}^{-1}{c}_i^T(\nabla V_{\mathrm{iF}}^{*})+\frac{1}{4}{(\nabla V_{\mathrm{iL}}^{*})}^T}\\ \multicolumn{1}{c}{g_i{R}_{\mathrm{iL}}^{-1}{g}_i^T(\nabla V_{\mathrm{iL}}^{*})+\frac{1}{4}{(\nabla V_{\mathrm{iL}}^{*})}^T{c}_i{R}_{\mathrm{iLi}}^{-1}{c}_i^T(\nabla V_{\mathrm{iL}}^{*})}\\ \multicolumn{1}{c}{+{\lambda }_i^T{\stackrel{·}{\hslash }}_{i1}+{(\nabla V_{\mathrm{iF}}^{*}+\nabla V_{\mathrm{iL}}^{*})}^T(\sum _{v=1}^n{G}_v{u}_v+C\left(x\right)).}\end{array}$$

(44)

Considering (44), one obtains:

$$\begin{array}{c}\multicolumn{1}{c}{{\stackrel{·}{V}}_{\mathrm{iLy}}\left(t\right)=-{\stackrel{·}{E}}^T(Q_{\mathrm{iF}}+Q_{\mathrm{iL}})\stackrel{·}{E}-{(\nabla V_{\mathrm{iF}}^{*})}^T\left(g_i(u_i^{*}-{\hat{u}}_i)\right)}\\ \multicolumn{1}{c}{-{(\nabla V_{\mathrm{iL}}^{*})}^T(c_i^{*}-{\hat{c}}_i)+\frac{1}{4}{(\nabla V_{\mathrm{iF}}^{*})}^T{g}_i{R}_{\mathrm{iF}}^{-1}{g}_i^T(\nabla V_{\mathrm{iF}}^{*})}\\ \multicolumn{1}{c}{+\frac{1}{4}{(\nabla V_{\mathrm{iF}}^{*})}^T{c}_i{R}_{\mathrm{iFi}}^{-1}{c}_i^T(\nabla V_{\mathrm{iF}}^{*})+\frac{1}{4}{(\nabla V_{\mathrm{iL}}^{*})}^T{g}_i{R}_{\mathrm{iL}}^{-1}{g}_i^T}\\ \multicolumn{1}{c}{(\nabla V_{\mathrm{iL}}^{*})+\frac{1}{4}{(\nabla V_{\mathrm{iL}}^{*})}^T{c}_i{R}_{\mathrm{iLi}}^{-1}{c}_i^T(\nabla V_{\mathrm{iL}}^{*})+{\lambda }_i^T{\stackrel{·}{\hslash }}_{i1}.}\end{array}$$

(45)

Then, substituting (41) into (45), we have:

$$\begin{array}{c}\multicolumn{1}{c}{{\stackrel{·}{V}}_{\mathrm{iLy}}\left(t\right)=-{\stackrel{·}{E}}^T(Q_{\mathrm{iF}}+Q_{\mathrm{iL}})\stackrel{·}{E}+\frac{1}{4}{(\nabla V_{\mathrm{iF}}^{*})}^T{g}_i{R}_{\mathrm{iF}}^{-1}{g}_i^T}\\ \multicolumn{1}{c}{(\nabla V_{\mathrm{iF}}^{*})+\frac{1}{4}{(\nabla V_{\mathrm{iF}}^{*})}^T{c}_i{R}_{\mathrm{iFi}}^{-1}{c}_i^T(\nabla V_{\mathrm{iF}}^{*})+\frac{1}{4}{(\nabla V_{\mathrm{iL}}^{*})}^T}\\ \multicolumn{1}{c}{g_i{R}_{\mathrm{iL}}^{-1}{g}_i^T(\nabla V_{\mathrm{iL}}^{*})+\frac{1}{4}{(\nabla V_{\mathrm{iL}}^{*})}^T{c}_i{R}_{\mathrm{iLi}}^{-1}{c}_i^T(\nabla V_{\mathrm{iL}}^{*})}\\ \multicolumn{1}{c}{+{\lambda }_i^T{\stackrel{·}{\hslash }}_{i1}+\frac{1}{2}{(\nabla {\varphi }_{\mathrm{iFcr}}\left(\stackrel{·}{E}\right){\hat{W}}_{\mathrm{iFcr}}+\nabla {\epsilon }_{\mathrm{iFcr}})}^T(g_i{R}_{\mathrm{iF}}^{-1}}\\ \multicolumn{1}{c}{(g_i^T\nabla {\varphi }_{\mathrm{iFcr}}{\tilde{W}}_{\mathrm{iFcr}}+g_i^T\nabla {\epsilon }_{\mathrm{iFcr}}))+\frac{1}{2}(\nabla {\varphi }_{\mathrm{iLcr}}\left(\stackrel{·}{E}\right){\hat{W}}_{\mathrm{iLcr}}}\\ \multicolumn{1}{c}{+\nabla {\epsilon }_{\mathrm{iLcr}}{)}^T\left(R_{\mathrm{iLi}}^{-1}(\nabla {\varphi }_{\mathrm{iLcr}}{\tilde{W}}_{\mathrm{iLcr}}+\nabla {\epsilon }_{\mathrm{iLcr}})\right)}\\ \multicolumn{1}{c}{=-{\stackrel{·}{E}}_T(Q_{\mathrm{iF}}+Q_{\mathrm{iL}})\stackrel{·}{E}+{\mathrm{\Pi }}_{\mathrm{iJ}},}\end{array}$$

(46)

in which ${\mathrm{\Pi }}_{\mathrm{iJ}}$ has up-bound:

$${\mathrm{\Pi }}_{\mathrm{iJ}}\le {\pi }_{\mathrm{iJ}},$$

(47)

where ${\pi }_{\mathrm{iJ}}$ is a constant.

Combining (46), ${\stackrel{·}{V}}_{\mathrm{iLy}}\left(t\right)$ has:

$$\begin{array}{c}\multicolumn{1}{c}{{\stackrel{·}{V}}_{\mathrm{iLy}}\left(t\right)\le -{\stackrel{·}{E}}_T(Q_{\mathrm{iF}}+Q_{\mathrm{iL}})\stackrel{·}{E}+{\pi }_{\mathrm{iJ}}}\\ \multicolumn{1}{c}{\le -{\lambda }_{\min }(Q_{\mathrm{iF}}+Q_{\mathrm{iL}})\parallel \stackrel{·}{E}{\parallel }^2+{\pi }_{\mathrm{iJ}}.}\end{array}$$

(48)

If $\stackrel{·}{E}$ lies outside:

$$\mathrm{\Omega }=\{\stackrel{·}{E}:\parallel \stackrel{·}{E}\parallel \le \sqrt{\frac{{\pi }_{\mathrm{iJ}}}{{\lambda }_{\min }(Q_{\mathrm{iF}}+Q_{\mathrm{iL}})}}\},$$

(49)

(41) is negative. Therefore, ${\stackrel{·}{V}}_{\mathrm{Ly}}\left(t\right)<0$ for any $\stackrel{·}{E}\ne 0$ when (49) is satisfied. The position tracking error under the coordinated operation task is UUB via hierarchical optimal coordinated control (39).

UUB means regardless of the complexity of the environment, and even in the presence of unavoidable disturbance and modeling error, the robot’s tracking error will not spiral out of control. It will ultimately be confined within a predetermined, acceptable range.

• Remark 4. In (8) and (9), matrices $Q_{\mathrm{iF}}$ and $Q_{\mathrm{iL}}$ affect the tightness of uniformly ultimately bounded. According to (49), the eigenvalue of matrices $Q_{\mathrm{iF}}$ and $Q_{\mathrm{iL}}$ is big, then the compact set is small, vice versa. Furthermore, the value of the selection matrix in (8) and (9) should be as close to the identity matrix as possible to reflect the relatively equal relationship among various controllers.

Experiments

Experiment 1 setup

This study evaluates the proposed hierarchical approximate optimal control strategy, grounded in the Nash-Stackelberg differential game, against a coordinated operation vertical moving task (Figure 2) on a 7-DOF MRM platform (Figure 3). The experiment is designed to fulfill dual control objectives: ensuring high-fidelity position tracking and minimizing control effort during coordinated manipulation. The proposed control algorithm, though initially developed in continuous time, necessitates discrete implementation during experimental procedures to enable real-time control. Specifically, the robotic control system is integrated within the Robot Operating System (ROS) framework on a host computer, which interfaces with the Sawyer controller to handle experimental data processing. This parameter sampling configuration substantially improves the online processing efficiency of the robotic system while conforming to real-time coding standards. In this experiment, the sampling frequency is set as 1000 Hz, that means controller processing 1000 sampling data every second.

Figure 2.

Experimental with coordinated operation task.

Figure 3.

Experimental platform setup.

Relevant system dynamics and controller configurations are provided in Table 1. The modular robot manipulator exhibits seven degree-of-freedom. Leveraging the forward computation characteristic of ADP, the curse of dimensionality is effectively circumvented. Each neural network, which comprises a single hidden layer with five neurons, entails a computational cost quantified by the number of operations performed per inference $O\left(5s_d\right)$ , where $s_d$ denotes the input dimension (e.g. 7 for position). The critic NN is instantiated using radial basis function neural network (RBFNN), where the activation function defined in (23) is ${\varphi }_c=e^{\frac{-{(\stackrel{·}{\mathrm{E}}-\psi )}^T(\stackrel{·}{\mathrm{E}}-\psi )}{\ell }}$ characterized by a predefined bandwidth $\psi$ and central vector $\ell$ .

Table 1.

Parameter definition.

Parameter type	Name	Value	Name	Value	Name	Value
Model parameters	${\hat{f}}_{\mathrm{ibv}}$	$0.012\mathrm{Nms}/\mathrm{rad}$	${\hat{f}}_{\mathrm{ico}}$	$0.03\mathrm{Nm}$	${\hat{f}}_{\mathrm{ist}}$	$0.04\mathrm{Nm}$
	${\hat{f}}_{i\tau s}$	$20s^2/{\mathrm{rad}}^2$	${\gamma }_{\mathrm{ira}}$	$100$	$I_{\mathrm{imo}}$	$0.12{\mathrm{kgcm}}^2$
Control parameters	$b_{\mathrm{ip}1}$	$0.08\mathrm{Nms}/\mathrm{rad}$	$b_{\mathrm{ip}2}$	$0.11\mathrm{mNm}$	$b_{\mathrm{ip}3}$	$0.18\mathrm{mNm}$
	$b_{\mathrm{ip}4}$	$63s^2/{\mathrm{rad}}^2$	$b_{\mathrm{ipd}}$	$0.71$	$b_{\mathrm{idc}}$	$6.83$
	${\varsigma }_{\mathrm{iLcr}}$	$0.99$	$n$	$7$	${\varsigma }_{\mathrm{iFcr}}$	$0.96$
	${\xi }_{\mathrm{iLL}}$	$8.63$	${\pi }_{\mathrm{iJ}}$	$10.06$	${\xi }_{\mathrm{iLF}}$	$7.23$
	$R_{\mathrm{iF}}$	$0.89I$	$R_{\mathrm{iFv}}$	$0.82I$	$Q_{\mathrm{iF}}$	$0.86I$
	$R_{\mathrm{iL}}$	$0.92I$	$R_{\mathrm{iLv}}$	$0.98I$	$Q_{\mathrm{iL}}$	$0.90I$

Experiment 1 results

A comparative evaluation is conducted to validate the proposed method, encompassing an existing learning-based optimal control approach without Nash-Stackelberg differential game (e.g. distributed optimal control via ADP^27,28) and the novel Nash-Stackelberg differential game-based hierarchical approximate optimal coordinated control framework introduced herein.

• (1) Position tracking

The experimental results, as presented in Figures 4 through 6, demonstrate a significant improvement in tracking accuracy within both Cartesian and joint spaces when utilizing the proposed Nash-Stackelberg game-theoretic coordinated control framework. Compared to conventional control method, the error magnitudes achieved under this approach are markedly lower. Importantly, although trajectory errors tend to increase when navigating sharp corners, the proposed strategy effectively constrains these deviations within a narrow margin during steady state operation. This robust performance is largely attributed to the effective compensation provided by the Stackelberg game.

Figure 4.

Position tracking in Cartesian space via existed optimal control (right) and developed control approach (left).

Figure 5.

First joint of position tracking via existed optimal control (lower) and developed control approach (upper).

Figure 6.

Joint position tracking error via existed optimal control (lower) and developed control approach (upper).

• (2) Control torque

Figure 7 illustrates the control torque responses observed during the coordinated operation task, employing both the existing and proposed Nash-Stackelberg differential game-based hierarchical approximate optimal coordinated control strategies. The proposed hierarchical control framework, rooted in the Nash-Stackelberg differential game, effectively maintains control torque transients within predefined safe operational limits. This capability stems from its inherent approximate optimal control mechanism, which is specifically designed to optimize output torques, thereby ensuring both stability and efficiency during task execution.

Figure 7.

Control torque via existed optimal control (lower) and developed control approach (upper).

• (3) Interaction torque

An analysis of interaction torque is presented in Figure 8. The proposed hierarchical control, leveraging the Nash-Stackelberg differential game, demonstrates superior torque regulation capability. It maintains interaction torque within a stable, reasonable bound and effectively attenuates chattering. This improvement stems from the game-theoretic mechanism’s ability to ensure operational security and seamlessly reconcile the dynamic coupling between the MRM and its task. Moreover, the integrated approximate optimal control further minimizes the torque magnitude, particularly during trajectory changing suddenly.

Figure 8.

Interaction torque curves via proposed control approach.

• (4) Critic NN weight

The evolution of the critic neural network weights, as depicted in Figure 9, reveals a direct correlation with the control torque. This observed dynamic correspondence confirms the NN’s capability to encode optimal control policies in real-time. The initial weights of critic NN are all 0 and learning rates are 0.99 and 0.96 for leader and followers. $\psi =[-3,-1.5,0,1.5,3],\ell =3.$

Figure 9.

First joint of critic NN weight curves via proposed control approach.

Experiment 2

To further validate the Nash-Stackelberg differential game-based approach to approximate optimal coordinated control, a more demanding experiment involving a complex trajectory is conducted. To verify the robustness of the proposed algorithm, collision is applied on MRM for around 22 s.

• (1) Position tracking

The experimental results in Figures 10 and 11 reveal a notable enhancement in tracking precision in Cartesian and joint space under the proposed Nash-Stackelberg game-theoretic coordinated control framework. Its error amplitude is substantially lower, while errors naturally increase at collision happened, the proposed scheme successfully confines them within a tight bound during steady-state phases, thanks to the robustness and effective compensation of the IDC dynamic.

• (2) Control torque

Figure 10.

Position tracking in Cartesian space via developed control approach.

Figure 11.

Joint position tracking error via developed control approach.

Figure 12 is control torque under coordinated operation task via proposed Nash-Stackelberg differential game-based hierarchical approximate optimal coordinated control. The proposed hierarchical control strategy, grounded in the Nash-Stackelberg differential game, ensures the restraint of control torque transients within safe operational bounds even collision happened. This performance is achieved through its inherent approximate optimal control mechanism, which explicitly addresses the optimization of output torques.

Figure 12.

Control torque via developed control approach.

• (3) Critic NN weight

Figure 13 presents the evolution of the critic neural network weights associated with joint 0. The weight parameters exhibit stable convergence within an acceptable range, thereby validating the feasibility of the proposed control approach.

Figure 13.

First joint of critic NN weight curves via proposed control approach.

Conclusion

This study develops a hierarchical approximate optimal coordinated control scheme for MRMs modeled via joint torque feedback, using ADP. Within the Nash-Stackelberg differential game framework, the coordinated control issue is reformulated as a Stackelberg differential game, where the MRM acts as the leader under a Nash differential game formulation and the coordination objective serves as the follower. The approximation of the nonlinear cost function is achieved via critic neural network, thereby deriving an optimal control strategy. Both rigorous theoretical analysis and comprehensive experimental results confirm the efficacy of the proposed control scheme. Computational complexity and scalability constraint continue to pose substantial obstacles to the practical deployment of adaptive dynamic programming and reinforcement learning methodologies in real-world application. This challenge is particularly pronounced in MRM systems engaged in coordinated manipulation tasks, where the demand for real-time computation intensifies. Achieving an optimal balance between high-precision task execution and computational efficiency in MRM system remains a critical challenge that requires immediate resolution.