A well-controlled active suspension system has the potential to provide better ride comfort. Benefiting from its powerful feature extraction and nonlinear generalization capabilities, the deep reinforcement learning (DRL), such as deep deterministic policy gradient (DDPG), has shown great potential to make decisions adaptively and intelligently in the control of active suspension system. However, the DDPG is troubled by the problem of low training efficiency due to the high proportion of illegal strategies. This paper proposed a novel DDPG controller for a nonlinear uncertain active suspension system by combining DRL with expert demonstrations. Specifically, the improved training method integrated with both a pre-training mechanism based on PID expert samples and an adaptive experience replay mechanism, is put forward for the DDPG to achieve both the goals of imitating the expert and improving the training efficiency. Moreover, considering the ride comfort and the state constraints as targets, a mixed reward function is designed to guide RL agents for learning effective actions. It is shown that the proposed training methods effectively accelerate the convergence of the DDPG. Furthermore, the comparison experiments demonstrate that the proposed controller provides great vibration attenuation, and has better adaptiveness to various working conditions and parametric uncertainty.
PangHZhangXChenJ, et al. Design of a coordinated adaptive backstepping tracking control for nonlinear uncertain active suspension system. Appl Math Model2019; 76: 479–494.
2.
HuangYNaJWuX, et al. Adaptive control of nonlinear uncertain active suspension systems with prescribed performance. ISA Trans2015; 54: 145–155.
3.
SavaresiSMSpeltaC. A single-sensor control strategy for semi-active suspensions. IEEE Trans Control Syst Technol2009; 17(1): 143–152.
4.
AghababaMP. A fractional-order controller for vibration suppression of uncertain structures. ISA Trans2013; 52(6): 881–887.
5.
HongFPangCK. Robust vibration control at critical resonant modes using indirect-driven self-sensing actuation in mechatronic systems. ISA Trans2012; 51(6): 834–840.
6.
PangHZhangXYangJ, et al. Adaptive backstepping-based control design for uncertain nonlinear active suspension system with input delay. Int J Robust Nonlinear Control2019; 29(16): 5781–5800.
7.
WangGChadliMBasinMV. Practical terminal sliding mode control of nonlinear uncertain active suspension systems with adaptive disturbance observer. IEEE ASME Trans Mechatron2020; 26(2): 789–797.
8.
DuHZhangN. Fuzzy control for nonlinear uncertain electrohydraulic active suspensions with input constraint. IEEE Trans Fuzzy Syst2008; 17(2): 343–356.
9.
WuXZhouBHuangJ, et al. Switched anti-roll control design for hydraulically interconnected suspension system with time delay. Int J Heavy Veh Syst2019; 79(2–3): 143–167.
10.
SavasYBasturkHI. Adaptive backstepping control design for active suspension systems actuated by four-way valve-piston. In: 2017 American control conference (ACC), Seattle, WA, USA, 24–26 May 2017, pp.438–443. New York: IEEE.
11.
SuX. Master–slave control for active suspension systems with hydraulic actuator dynamics. IEEE Access2017; 5: 3612–3621.
12.
KonoikoAKadhemASaifulI, et al. Deep learning framework for controlling an active suspension system. J Vib Control2019; 25(17): 2316–2329.
13.
LiuYJZengQLiuL, et al. An adaptive neural network controller for active suspension systems with hydraulic actuator. IEEE Trans Syst Man Cybern Syst2018; 50(12): 5351–5360.
14.
JingHLiXKarimiHR. Output-feedback-based H∞ control for vehicle suspension systems with control delay. IEEE Trans Ind Electron2014; 61(1): 436–446.
15.
LianRJ. Enhanced adaptive self-organizing fuzzy sliding-mode controller for active suspension systems. IEEE Trans Ind Electron2013; 60(3): 958–968.
16.
LiHLiuHGaoH, et al. Reliable fuzzy control for active suspension systems with actuator delay and fault. IEEE Trans Fuzzy Syst2012; 20(2): 342–357.
17.
SandeTGysenBBesselinkI, et al. Robust control of an electromagnetic active suspension system: simulations and measurements. Mechatronics2013; 23(2): 204–212.
18.
DangorMDahunsiOAPedroJO, et al. Evolutionary algorithm-based PID controller tuning for nonlinear quarter-car electrohydraulic vehicle suspensions. Nonlinear Dyn2012; 78(4): 2795–2810.
19.
SunWGaoHKaynakO. Adaptive backstepping control for active suspension systems with hard constraints. IEEE ASME Trans Mechatron2013; 18(3): 1072–1079.
20.
ZhangJLiKLiY. Neuro-adaptive optimized control for full active suspension systems with full state constraints. Neurocomputing2021; 458: 478–489.
21.
ZhangYLiuYWangZ, et al. Neural networks-based adaptive dynamic surface control for vehicle active suspension systems with time-varying displacement constraints. Neurocomputing2020; 408: 176–187.
22.
PangHZhangXXuZ. Adaptive backstepping-based tracking control design for nonlinear active suspension system with parameter uncertainties and safety constraints. ISA Trans2019; 88: 23–36.
23.
MnihVKavukcuogluKSilverD, et al. Human-level control through deep reinforcement learning. Nature2015; 518(7540): 529–533.
24.
MnihVBadiaAPMirzaM, et al. Asynchronous methods for deep reinforcement learning. In: International conference on machine learning, New York, NY, USA, June 2016, pp.1928–1937. PMLR.
LillicrapTPHuntJPritzelA, et al. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
27.
HaarnojaTZhouAAbbeelP, et al. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: International conference on machine learning, Vienna, Austria, July 2018, pp.1861–1870. PMLR.
28.
ZhaoJChengSLiL, et al. A model free controller based on reinforcement learning for active steering system with uncertainties. Proc IMechE, Part D: J Automobile Engineering2021; 235(9): 2470–2483.
29.
WangHYuanSGuoM, et al. Tactical driving decisions of unmanned ground vehicles in complex highway environments: a deep reinforcement learning approach. Proc IMechE, Part D: J Automobile Engineering2021; 235(4): 1113–1127.
30.
AradiS. Survey of deep reinforcement learning for motion planning of autonomous vehicles. IEEE Trans Intell Transp Syst2020; 23(2): 740–759.
31.
KiranBRSobhITalpaertV, et al. Deep reinforcement learning for autonomous driving: a survey. IEEE Trans Intell Transp Syst2021; 23(6): 4909–4926.
32.
TaylorPBucakİÖÖzHR. Vibration control of a nonlinear quarter-car active suspension system by reinforcement learning. Int J Syst Sci2012; 43(6): 1177–1190.
33.
ChenHCLinYCChangYH. An actor-critic reinforcement learning control approach for discrete-time linear system with uncertainty. In: 2018 international automatic control conference (CACS), Taoyuan, Taiwan, 4–7 November 2018, pp.1–5. New York: IEEE.
34.
LiZChuTKalabićU. Dynamics-enabled safe deep reinforcement learning: case study on active suspension control. In: 2019 IEEE conference on control technology and applications (CCTA), Hong Kong, China, 19–21 August 2019, pp.585–591. New York: IEEE.
35.
FaresAYounesAB. Online reinforcement learning-based control of an active suspension system using the actor critic approach. Appl Sci2020; 10(22): 8060.
36.
LinYCNguyenHLTYangJF, et al. A reinforcement learning backstepping-based control design for a full vehicle active Macpherson suspension system. IET Control Theory Appl2022; 16: 1417–1430.
37.
HuangZWuJLvC. Efficient deep reinforcement learning with imitative expert priors for autonomous driving. IEEE Trans Neural Netw Learn Syst. Epub ahead of print 26January2022. DOI: 10.1109/TNNLS.2022.3142822.
38.
ScheelOBergaminiLWolczykM, et al. Urban driver: learning to drive from real-world demonstrations using policy gradients. In: Conference on robot learning, Paris, France, April 2022, pp.718–728. PMLR.
39.
LiuHHuangZLvC. Improved deep reinforcement learning with expert demonstrations for urban autonomous driving. arXiv preprint arXiv:2102.09243, 2021.
40.
ChuZSunBZhuD, et al. Motion control of unmanned underwater vehicles via deep imitation reinforcement learning algorithm. IEEE Trans Intell Transp Syst2020; 14(7): 764–774.
41.
SunWPanHGaoH. Filter-based adaptive vibration control for active vehicle suspensions with electrohydraulic actuators. IEEE Trans Veh Technol2015; 65(6): 4619–4626.
MunawwarahSYakubF. Control analysis of vehicle ride comfort through integrated control devices on the quarter and half car active suspension systems. Proc IMechE, Part D: J Automobile Engineering2021; 235(5): 1256–1268.
44.
LuXLiaoWHuangW, et al. An improved linear quadratic regulator control method through convolutional neural network–based vibration identification. J Vib Control2021; 27(7–8): 839–853.
45.
ISO 2631-1:1997. Mechanical vibration and shock—evaluation of human exposure to whole-body vibration—part 1: general requirements. International Organization for Standardization.
HuHFazlyabMMorariM, et al. Reach-SDP: reachability analysis of closed-loop systems with neural network controllers via semidefinite programming. In: 2020 59th IEEE conference on decision and control (CDC), Seogwipo, Korea, 14–18 December 2020, pp.5929–5934. New York: IEEE.
