Benefitting from the technology of integral reinforcement learning, the nonzero sum (NZS) game for distributed parameter systems is effectively solved in this paper when the information of system dynamics are unavailable. The Karhunen-Loève decomposition (KLD) is employed to convert the partial differential equation (PDE) systems into high-order ordinary differential equation (ODE) systems. Moreover, the off-policy IRL technology is introduced to design the optimal strategies for the NZS game. To confirm that the presented algorithm will converge to the optimal value functions, the traditional adaptive dynamic programming (ADP) method is first discussed. Then, the equivalence between the traditional ADP method and the presented off-policy method is proved. For implementing the presented off-policy IRL method, actor and critic neural networks are utilized to approach the value functions and control strategies in the iteration process, individually. Finally, a numerical simulation is shown to illustrate the effectiveness of the proposal off-policy algorithm.
AksikasIFuxmanAForbesJFWinkinJJ (2009) LQ control design of a class of hyperbolic PDE systems: Application to fixed-bed reactor. Automatica45(6): 1542–1548.
2.
AksikasIWinkinJJDochainD (2007) Optimal LQ-feedback regulation of a nonisothermal plug flow reactor model by spectral factorization. IEEE Transactions on Automatic Control52(7): 1179–1193.
3.
ArmaouAChristofidesPD (2001a) Crystal temperature control in the Czochralski crystal growth process. AIChE Journal47(1): 79–106.
4.
ArmaouAChristofidesPD (2001b) Finite-dimensional control of nonlinear parabolic PDE systems with time-dependent spatial domains using empirical eigenfunctions. International Journal of Applied Mathematics and Computer Science11: 287–317.
5.
ArmaouAChristofidesPD (2002) Dynamic optimization of dissipative PDE systems using nonlinear order reduction. Chemical Engineering Science57(24): 5083–5114.
6.
BakerJChristofidesPD (1999) Output feedback control of parabolic PDE systems with nonlinear spatial differential operators. Industrial and Engineering Chemistry Research38(11): 4372–4380.
7.
ChristofidesPD (1998) Robust control of parabolic PDE systems. Chemical Engineering Science53(16): 2949–2965.
8.
CurtainRFZwartH (2012) An Introduction to Infinite-dimensional Linear Systems Theory, volume 21. New York: Springer Science and Business Media.
9.
FuJBaiJLaiJLiPYuMLamHK (2019) Adaptive fuzzy control of a magnetorheological elastomer vibration isolation system with time-varying sinusoidal excitations. Journal of Sound and Vibration456: 386–406.
10.
KalaniAChristofidesPD (2002) Simulation, estimation and control of size distribution in aerosol processes with simultaneous reaction, nucleation, condensation and coagulation. Computers and Chemical Engineering26(7–8): 1153–1169.
11.
LeakeRLiuRW (1967) Construction of suboptimal control sequences. SIAM Journal on Control5(1): 54–63.
12.
LewisFLVamvoudakisKG (2010) Reinforcement learning for partially observable dynamic processes: Adaptive dynamic programming using measured output data. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)41(1): 14–25.
13.
LewisFLVrabieD (2009) Reinforcement learning and adaptive dynamic programming for feedback control. IEEE Circuits and Systems Magazine9(3): 32–50.
14.
LiMChristofidesPD (2007) An input/output approach to the optimal transition control of a class of distributed chemical reactors. Chemical Engineering Science62(11): 2979–2988.
15.
LiangHZhangLSunYHuangT (2019) Containment control of semi-Markovian multiagent systems with switching topologies. IEEE Transactions on Systems, Man, and Cybernetics: Epub ahead of print. DOI: 10.1109/TSMC.2019.2946248.
16.
LiangHZhouYMaHZhouQ (2018) Adaptive distributed observer approach for cooperative containment control of nonidentical networks. IEEE Transactions on Systems, Man, and Cybernetics: Systems49(2): 299–307.
17.
LiuDWangDWangFYLiHYangX (2014) Neural-network-based online HJB solution for optimal robust guaranteed cost control of continuous-time uncertain nonlinear systems. IEEE Transactions on Cybernetics44(12): 2834–2847.
18.
LuoBWuHN (2012a) Approximate optimal control design for nonlinear one-dimensional parabolic PDE systems using empirical eigenfunctions and neural network. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)42(6): 1538–1549.
19.
LuoBWuHN (2012b) Online policy iteration algorithm for optimal control of linear hyperbolic PDE systems. Journal of Process Control22(7): 1161–1170.
20.
LuoBHuangTWuHNYangX (2015) Data-driven control for nonlinear distributed parameter systems. IEEE Transactions on Neural Networks and Learning Systems26(11): 2949–2961.
21.
LuoBWuHNHuangTLiuD (2014) Data-based approximate policy iteration for affine nonlinear continuous-time optimal control design. Automatica50(12): 3281–3290.
22.
MuYZhangHSuHZhangK (2019a) Observer-based actuator fault estimation and proportional derivative fault tolerant control for continuous-time singular systems. Optimal Control Applications and Methods40(6): 979–997.
23.
MuYZhangHSunSRenJ (2019b) Robust non-fragile proportional plus derivative state feedback control for a class of uncertain Takagi–Sugeno fuzzy singular systems. Journal of the Franklin Institute356(12): 6208–6225.
24.
PadhiRBalakrishnanS (2003) Proper orthogonal decomposition based optimal neurocontrol synthesis of a chemical reactor process using approximate dynamic programming. Neural Networks16(5–6): 719–728.
25.
PitchaiahSArmaouA (2010) Output feedback control of distributed parameter systems using adaptive proper orthogonal decomposition. Industrial and Engineering Chemistry Research49(21): 10496–10509.
26.
PitchaiahSArmaouA (2013) Output feedback control of dissipative PDE systems with partial sensor information based on adaptive model reduction. AIChE Journal59(3): 747–760.
27.
RayW (1981) Advanced Process Control. New York: McGraw-Hill.
28.
RenHZhangHWenYLiuC (2019) Integral reinforcement learning off-policy method for solving nonlinear multi-player nonzero-sum games with saturated actuator. Neurocomputing335: 96–104.
29.
SadekIBokhariM (1998) Optimal control of a parabolic distributed parameter system via orthogonal polynomials. Optimal Control Applications and Methods19(3): 205–213.
30.
SongRLewisFLWeiQZhangH (2015) Off-policy actor-critic structure for optimal control of unknown systems with disturbances. IEEE Transactions on Cybernetics46(5): 1041–1050.
31.
SuttonRSBartoAG (1998) Reinforcement Learning: An Introduction, volume 116. Cambridge, MA, USA: Cambridge University Press.
WuHNLuoB (2014) L2 disturbance attenuation for highly dissipative nonlinear spatially distributed processes via HJI approach. Journal of Process Control24(5): 550–567.
35.
YadavVPadhiRBalakrishnanS (2007) Robust/optimal temperature profile control of a high-speed aerospace vehicle using neural networks. IEEE Transactions on Neural Networks18(4): 1115–1128.
36.
ZhangHCuiLLuoY (2012a) Near-optimal control for nonzero-sum differential games of continuous-time nonlinear systems using single-network ADP. IEEE Transactions on Cybernetics43(1): 206–216.
37.
ZhangHLiuDLuoYWangD (2012b) Adaptive Dynamic Programming for Control: Algorithms and Stability. London, UK: Springer Science and Business Media.
38.
ZhangHWeiQLiuD (2011) An iterative adaptive dynamic programming method for solving a class of nonlinear zero-sum differential games. Automatica47(1): 207–214.
39.
ZhangJChenXGuG (2020) State consensus for discrete-time multi-agent systems over time-varying graphs. arXiv: Optimization and Control. Epub ahead of print. DOI: 10.1109/TAC.2020.2979750
40.
ZhangJWangZZhangH (2018) Data-based optimal control of multiagent systems: A reinforcement learning design approach. IEEE Transactions on Cybernetics49(12): 4441–4449.
41.
ZhangJZhangHFengT (2017a) Distributed optimal consensus control for nonlinear multiagent system with unknown dynamic. IEEE Transactions on Neural Networks and Learning Systems29(8): 3339–3348.
42.
ZhangQZhaoDZhuY (2017b) Data-driven adaptive dynamic programming for continuous-time fully cooperative games with partially constrained inputs. Neurocomputing238: 377–386.
