The mixed dual-performance tracking control optimization problem for continuous-time systems with unknown internal dynamics is solved in the framework of Stackelberg games with hierarchical control. Compared with the traditional Nash games, the mixed tracking optimization problem is solved by finding the solution to a two-stage constrained optimal tracking control problem. The hierarchical tracking control problem is first changed into an optimization regulation problem by augmenting the state vector. The control input and the unknown external disturbances are deemed as the leader and the follower, which is obtained by solving the coupled hierarchical leader–follower Hamiltonian–Jacobi (HJ) tracking equations with constrained conditions. An improved hierarchical online integral reinforcement learning (IRL) tracking control algorithm is proposed to obtain the Stackelberg equilibrium and the optimal cost functions iteratively based on the Lyapunov-like equations. The single NN structure is applied for each player to approximate the unknown cost function. In addition, the sufficient and necessary condition is discussed about the uniqueness and existence of Stackelberg equilibrium. The universality and the practicability of the learning algorithm are shown by two simulation results.
BagchiABasarT (1981) Stackelberg strategies in linear-quadratic stochastic differential games. Journal of Optimization Theory and Applications35(3): 433–464.
2.
BernsteinDHaddadW (1989) LQG control with an H∞ performance bound A Riccati equation approach. IEEE Transactions on Automatic Control34(3): 293–305.
3.
ChenZTChenKRChenSZ, et al. (2022) Event-triggered H∞ consensus for uncertain nonlinear systems using integral sliding mode based adaptive dynamic programming. Neural Networks156: 258–270.
4.
CuiXHChenJYWangBR, et al. (2022) Off-policy algorithm based Hierarchical optimal control for completely unknown dynamic systems. Neurocomputing448: 669–680.
5.
CuiXHWangBRWangLN, et al. (2021) Online optimal learning algorithm for Stackelberg games with partially unknown dynamics and constrained inputs. Neurocomputing445: 1–11.
6.
FreilingGJankGAbou-KandilH (1999) Discrete-time Riccati equations in open-loop Nash and Stackelberg games. European Journal of Control5(1): 56–66.
7.
HamidrezaMLewisFLJiangZP, et al. (2015) H∞ tracking control of completely unknown continuous-time systems via off-policy reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems26(10): 2550–2562.
8.
HuangYZLiuDR (2014) Neural-network-based optimal tracking control scheme for a class of unknown discrete-time nonlinear systems using iterative ADP algorithm. Neurocomputing125: 46–56.
9.
JoseBCruzJ (1978) Leader-follower strategies for multilevel systems. IEEE Transactions on Automatic Control23(2): 244–255.
10.
KhargonekarPRoteaM (1991) Mixed H2/H∞ control: A convex optimization approach. IEEE Transactions on Automatic Control36(7): 824–837.
11.
LiMQinJFrerisM (2022) Multiplayer Stackelberg-Nash game for nonlinear system via value iteration-based integral reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems33(4): 1429–1440.
12.
LiXQWangWXuJG, et al. (2019) Stackelberg game approach to mixed H2/H∞ problem for continuous-time system. Journal of Systems Science and Complexity32: 1324–1339.
13.
LiXQXuJWangW (2018) Mixed H2/H∞ control for discrete-time systems with input delay. IET Control Theory and Applications12(16): 2221–2031.
14.
LiXQXuJZhangHS (2020) Standard solution to mixed H2/H∞ control with regular Riccati equation. IET Control Theory and Applications14(20): 3643–3651.
15.
LimebeerDAnersonBHendelB (1994) A Nash game approach to mixed H2/H∞ control. IEEE Transactions on Automatic Control39(1): 69–82.
16.
LiuDWeiQ (2014) Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems. IEEE Transactions on Neural Networks and Learning Systems25(3): 621–634.
17.
LiuDRLiHWangD (2014) Online synchronous approximate optimal learning algorithm for multiplayer nonzero-sum games with unknown dynamics. IEEE Transactions on Systems, Man, and Cybernetics44(8): 1015–1027.
18.
LiuPSunJZhangH, et al. (2024) Combination therapy-based adaptive control for organism using medicine dosage regulation mechanism. IEEE Transactions on Cybernetics54(1): 199–208.
19.
LiuPDLiuZMAoWA, et al. (2023) Dynamic event-triggered adaptive control for zero-sum games of nonlinear stochastic systems. Transactions of the Institute of Measurement and Control. Epub ahead of print 18February. DOI: 10.1177/01423312221 149776.
20.
MarcusJRushikeshKShubhenduB, et al. (2015) Approximate N-player nonzero-sum game solution for an uncertain continuous nonlinear system. IEEE Transactions on Neural Networks and Learning Systems26(8): 1645–1658.
21.
MedanicJ (1978) Closed-loop Stackelberg strategies in linear-quadratic problems. IEEE Transactions on Automatic Control23(4): 632–637.
22.
MingZNZhangHGTongX, et al. (2022) Mixed H2/H∞ control with dynamic event-triggered mechanism for partially unknown nonlinear stochastic systems. IEEE Transactions on Automation Science and Engineering20: 1934–1944.
23.
MuCXWangKZhangQC (2020) Hierarchical optimal control for input-affine nonlinear systems through the formulation of Stackelberg game. Information Sciences517: 1–17.
24.
MuCXZhangYGaoZK, et al. (2020) ADP-based robust tracking control for a class of nonlinear systems with unmatched uncertainties. IEEE Transactions on Systems, Man, and Cybernetics50: 4056–4067.
25.
MukaidaniH (2013) H2/H∞ control of stochastic systems with multiple decision makers: A Stackelberg game approach. In: IEEE 52nd annual conference on decision and control, Firenze, 10–13 December 2013, pp. 1750–1755. New York: IEEE.
26.
MukaidaniH (2014) Stackelberg strategy for discrete-time stochastic system and its application to H2/H∞ control. In: Proceeding of the American Control Conference, Portland, OR, 4–6 June, pp. 4488–4493. New York: IEEE.
27.
MuradoreRPicciG (2005) Mixed H2/H∞ control: The discrete-time case. Systems and Control Letters54(1): 1–13.
28.
PuJMaQGuF, et al. (2018) Data-driven adaptive dynamic programming for H2/H∞ control of unknown nonlinear system. In: 37th Chinese Control Conference (CCC), Wuhan, China, 25–27 July 2018, pp. 971–976.
29.
QinCBShangZZhangZ, et al. (2022) Robust tracking control for non-zero-sum games of continuous-time uncertain nonlinear systems. Mathematics10: 1904.
30.
QuanYZhangH (2011) Modeling, identification and control of a class of nonlinear system. Control Theory and Applications9(2): 349–354.
31.
ShengLZhangWGaoM (2014) Mixed H2/H∞ control of time-varying stochastic discrete-time systems under uniform detectability. IET Control Theory and Applications8(17): 1866–1874.
32.
VamvoudakisKGLewisFL (2010) Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica46: 878–888.
VamvoudakisKGLewisFLDixonWE (2019) Open-loop Stackelberg learning solution for hierarchical control problems. International Journal of Adaptive Control and Signal Processing33: 285–299.
35.
VamvoudakisKGLewisFLJohnsonM (2012) Online learning algorithm for Stackelberg games in problems with hierarchy. In: IEEE 51st IEEE Conference on Decision and Control (CDC), Maui, HI, 10–13 December 2012, pp. 1883–1889. New York: IEEE.
36.
VrabieDLewisFL (2009) Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems. Neural Networks22: 237–246.
37.
WeiQLZhuLSongRZ, et al. (2022) Model-free adaptive optimal control for unknown nonlinear multiplayer nonzero-sum game. IEEE Transactions on Neural Networks and Learning Systems33(2): 879–892.
38.
WuHNLuoB (2012) Neural network based online simultaneous policy update algorithm for solving the HJI equation in nonlinear H∞ control. IEEE Transactions on Neural Networks and Learning Systems23(12): 1884–1895.
39.
XieLLuLZhangD, et al. (2004) Improved robust H2 and H∞ filtering for uncertain discrete-time systems. Automatica40(5): 873–880.
40.
YangYWanYZhuJ, et al. (2021) Tracking control for linear discrete-time systems: Model-free Q-learning designs. IEEE Control Systems Letters5: 175–180.
41.
YangYLMazouchiMModaresH (2021) Hamiltonian-driven adaptive dynamic programming for mixed H2/H∞ performance using sum-of-squares. International Journal of Robust and Nonlinear Control31(6): 1941–1963.
42.
ZhanYLiuCHZhaoY (2020) Free market of multi-leader multi-follower mobile crowdsensing: An incentive mechanism design by deep reinforcement learning. IEEE Transactions on Mobile Computing19(10): 2316–2329.
43.
ZhangHGCuiLLZhangX, et al. (2011) Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method. IEEE Transactions on Neural Networks22(12): 2226–2236.
44.
ZhangHGCuiXHLuoYH (2018) Finite-horizon tracking control for unknown nonlinear systems with saturating actuators. IEEE Transactions on Neural Networks and Learning Systems29(4): 1200–1212.
45.
ZhangHGMingZNYanYQ, et al. (2021) Data-driven finite-horizon H∞ tracking control with event-triggered mechanism for the continuous-time nonlinear systems. IEEE Transactions on Neural Networks and Learning Systems34: 4687–4701.
46.
ZhangQZhaoD (2019) Data-based reinforcement learning for nonzero-sum games with unknown drift dynamics. IEEE Transactions on Cybernetics49: 2874–2885.
47.
ZhouLShenZP (2023) Event-triggered adaptive dynamic programming for optimal tracking control of unmanned surface vessel with input constraints. Transactions of the Institute of Measurement and Control45: 2835–2847.
48.
ZhuHZhangCSunP (2011) A Stackelberg game approach to mixed H2/H∞ robust control for singular bilinear systems. Trans Tech Publications204(210): 1839–1847.
