This paper proposes a model-data-driven control method for a human-leading vehicle platoon, comprising a human-driven vehicle (HDV) as the leader and connected automated vehicles (CAVs) as followers. Initially, a representative trajectory of HDVs is constructed using principal component analysis and the K-means clustering algorithm, which is utilized as training dataset. Subsequently, we propose a novel platooning method, named deep reinforcement learning with model-based guidance (DRLMG). The output of model predictive control (MPC) is integrated into the input state and reward function of the deep reinforcement learning (DRL) algorithm. The DRL algorithm benefits from guidance provided by MPC, leading to more optimal decision-making. To ensure safety and stability, a safety filter is designed using control barrier function and the control Lyapunov function. Simulation experiments with real-world driving data show that DRLMG outperforms MPC, reducing speed error, spacing error, and acceleration change rate by 17.9%, 53.7%, and 47.1%, respectively. In comparison to pure DRL, DRLMG increases spacing error by 6.5% but reduces speed error by 15.4% and acceleration change rate by 14.3%. The proposed method enhances DRL’s generalization capability, dampens traffic oscillations caused by the leading HDV, and guarantees driving safety and stability.
HuangZChuDWuC, et al. Path planning and cooperative control for automated vehicle platoon using hybrid automata. IEEE Trans Intell Transp Syst2019; 20(3): 959–974.
2.
BansalKMukhijaP. Event-triggered control of vehicular platoon system with time-varying delay and sensor faults. Proc IMechE, Part D: J Automobile Engineering2020; 234(14): 3362–3372.
3.
YinYHuangXZhanS, et al. Hierarchical model predictive control strategy based on Q-learning algorithm for hybrid electric vehicle platoon. Proc IMechE, Part D: J Automobile Engineering2022; 238(2–3): 385–402.
4.
WangJZhengYXuQ, et al. Controllability analysis and optimal control of mixed traffic flow with human-driven and autonomous vehicles. IEEE Trans Intell Transp Syst2021; 22(12): 7445–7459.
5.
RobinsonTChanECoelinghE. Operating platoons on public motorways: an introduction to the SARTRE platooning programme. In: 17th world congress on intelligent transport systems, Busan, Korea, 25–29 October 2010, vol. 1, p.12. Melbourne: ITS Asia Pacific.
6.
TsugawaSKatoSAokiK. An automated truck platoon for energy saving. In: 2011 IEEE/RSJ international conference on intelligent robots and systems, San Francisco, CA, USA, 25–30 September 2011, pp.4109–4114. New York: IEEE.
7.
LeeYAhnTLeeC, et al. A novel path planning algorithm for truck platooning using V2V communication. Sensors2020; 20(24): 7022.
8.
OzkanMFMaY. Distributed stochastic model predictive control for human-leading heavy-duty truck platoon. IEEE Trans Intell Transp Syst2022; 23(9): 16059–16071.
9.
LanJZhaoDTianD. Data-driven robust predictive control for mixed vehicle platoons using noisy measurement. IEEE Trans Intell Transp Syst2023; 24(6): 6586–6596.
10.
ZhengFLiuCLiuX, et al. Analyzing the impact of automated vehicles on uncertainty and stability of the mixed traffic flow. Transp Res Part C Emerg Technol2020; 112: 203–219.
11.
GeJIOroszG. Connected cruise control among human-driven vehicles: experiment-based parameter estimation and optimal control design. Transp Res Part C Emerg Technol2018; 95: 445–459.
12.
ZhangYTianBXuZ, et al. A local traffic characteristic based dynamic gains tuning algorithm for cooperative adaptive cruise control considering wireless communication delay. Transp Res Part C Emerg Technol2022; 142: 103766.
13.
GongSDuL. Cooperative platoon control for a mixed traffic flow including human drive vehicles and connected and autonomous vehicles. Transp Res Part B Methodol2018; 116: 25–61.
14.
FengSSongZLiZ, et al. Robust platoon control in mixed traffic flow based on tube model predictive control. IEEE Trans Intell Veh2021; 6(4): 711–722.
15.
ZhangYWuZZhangY, et al. Human-lead-platooning cooperative adaptive cruise control. IEEE Trans Intell Transp Syst2022; 23(10): 18253–18272.
16.
LiMCaoZLiZ. A reinforcement learning-based vehicle platoon control strategy for reducing energy consumption in traffic oscillations. IEEE Trans Neural Netw Learn Syst2021; 32(12): 5309–5322.
17.
ShiHZhouYWuK, et al. Connected automated vehicle cooperative control with a deep reinforcement learning approach in a mixed traffic environment. Transp Res Part C Emerg Technol2021; 133: 103421.
18.
JiangLXieYEvansNG, et al. Reinforcement learning based cooperative longitudinal control for reducing traffic oscillations and improving platoon stability. Transp Res Part C Emerg Technol2022; 141: 103744.
19.
WangZJinSLiuL, et al. Design of intelligent connected cruise control with vehicle-to-vehicle communication delays. IEEE Trans Veh Technol2022; 71(8): 9011–9025.
20.
ZhouYAhnSWangM, et al. Stabilizing mixed vehicular platoons with connected automated vehicles: an H-infinity approach. Transp Res Part B Methodol2020; 132: 152–170.
21.
YaoZHuRWangY, et al. Stability analysis and the fundamental diagram for mixed connected automated and human-driven vehicles. Phys A Stat Mech Appl2019; 533: 121931.
22.
ZhanZWangSMPanTL, et al. Stabilizing vehicular platoons mixed with regular human-piloted vehicles: an input-to-state string stability approach. Transportmetr B Transp Dyn2021; 9(1): 569–594.
23.
WangM. Infrastructure assisted adaptive driving to stabilise heterogeneous vehicle strings. Transp Res Part C Emerg Technol2018; 91: 276–295.
24.
CoppolaALuiDGPetrilloA, et al. Cooperative driving of heterogeneous uncertain nonlinear connected and autonomous vehicles via distributed switching robust PID-like control. Inf Sci2023; 625: 277–298.
25.
YinJShenDDuX, et al. Distributed stochastic model predictive control with Taguchi’s robustness for vehicle platooning. IEEE Trans Intell Transp Syst2022; 23(9): 15967–15979.
26.
NieGXieBHaoZ, et al. A distributed model predictive control approach to cooperative adaptive cruise control of the heterogeneous platoon. Proc IMechE, Part D: J Automobile Engineering2021; 236(14): 3153–3167.
27.
BansalKMukhijaP. Event-triggered control of interconnected nonlinear systems subjected to cyber-attacks and time-varying coupling. J Franklin Inst2022; 359(17): 9703–9733.
28.
AlbeaikSWuTVurimiG, et al. Deep truck cruise control: field experiments and validation of heavy duty truck cruise control using deep reinforcement learning. Control Eng Pract2022; 121: 105026.
29.
ZhangXPengYPanW, et al. Barrier function-based safe reinforcement learning for formation control of mobile robots. In: 2022 international conference on robotics and automation (ICRA), Philadelphia, PA, USA, 23–27 May 2022, pp.5532–5538. New York: IEEE.
30.
PrathibaSBRajaGDevK, et al. A hybrid deep reinforcement learning for autonomous vehicles smart-platooning. IEEE Trans Veh Technol2021; 70(12): 13340–13350.
31.
YanRJiangRJiaB, et al. Hybrid car-following strategy based on deep deterministic policy gradient and cooperative adaptive cruise control. IEEE Trans Autom Sci Eng2022; 19(4): 2816–2824.
32.
LiuTLeiLZhengK, et al. Autonomous platoon control with integrated deep reinforcement learning and dynamic programming. IEEE Internet Things J2023; 10(6): 5476–5489.
33.
MaHChenJEbenS, et al. Model-based constrained reinforcement learning using generalized control barrier function. In: 2021 IEEE/RSJ international conference on intelligent robots and systems (IROS), Prague, Czech Republic, 27 September–1 October 2021, pp.4552–4559. New York: IEEE.
34.
WangX. Ensuring safety of learning-based motion planners using control barrier functions. IEEE Robot Autom Lett2022; 7(2): 4773–4780.
35.
BrunkeLGreeffMHallAW, et al. Safe learning in robotics: from learning-based control to safe reinforcement learning. Annu Rev Control Robot Auton Syst2022; 5(1): 411–444.
36.
AmesADGrizzleJWTabuadaP. Control barrier function based quadratic programs with application to adaptive cruise control. In: 53rd IEEE conference on decision and control, Los Angeles, CA, USA, 15–17 December 2014, pp.6271–6278. New York: IEEE.
37.
ChenYSingletaryAAmesAD. Guaranteed obstacle avoidance for multi-robot operations with limited actuation: a control barrier function approach. IEEE Control Syst Lett2021; 5(1): 127–132.
38.
AmesADCooganSEgerstedtM, et al. Control barrier functions: theory and applications. In: 2019 18th European control conference (ECC), Naples, Italy, 25–28 June 2019, pp.3420–3431. New York: IEEE.
39.
AmesADXuXGrizzleJW, et al. Control barrier function based quadratic programs for safety critical systems. IEEE Trans Autom Control2017; 62(8): 3861–3876.
40.
GangopadhyayBDasguptaPDeyS. Safe and stable RL (S2RL) driving policies using control barrier and control Lyapunov functions. IEEE Trans Intell Veh2023; 8(2): 1889–1899.
41.
ChuTKalabićU. Model-based deep reinforcement learning for CACC in mixed-autonomy vehicle platoon. In: 2019 IEEE 58th conference on decision and control (CDC), Nice, France, 11–13 December 2019, pp.4079–4084. New York: IEEE.
42.
KherroubiZAknineSBachaR. Novel decision-making strategy for connected and autonomous vehicles in highway on-ramp merging. IEEE Trans Intell Transp Syst2022; 23(8): 12490–12502.
43.
KrajewskiRBockJKloekerL, et al. The highD dataset: a drone dataset of naturalistic vehicle trajectories on German highways for validation of highly automated driving systems. In: 2018 21st international conference on intelligent transportation systems (ITSC), Maui, HI, USA, 4–7 November 2018, pp.2118–2125. New York: IEEE.
44.
ZhaoXZhaoXYuQ, et al. Development of a representative urban driving cycle construction methodology for electric vehicles: a case study in Xi’an. Transp Res Part D Transp Environ2020; 81: 102279.
45.
JingZWangGZhangS, et al. Building Tianjin driving cycle based on linear discriminant analysis. Transp Res Part D Transp Environ2017; 53: 78–87.
46.
QiuHCuiSWangS, et al. A clustering-based optimization method for the driving cycle construction: a case study in Fuzhou and Putian, China. IEEE Trans Intell Transp Syst2022; 23(10): 18681–18694.
47.
DengZChuDWuC, et al. A probabilistic model for drivingstyle-recognition-enabled driver steering behaviors. IEEE Trans Syst Man Cybern Syst2022; 52(3): 1838–1851.
48.
ShenPZhaoZLiJ, et al. Development of a typical driving cycle for an intra-city hybrid electric bus with a fixed route. Transp Res Part D Transp Environ2018; 59: 346–360.
49.
LiuYZhouBWangX, et al. Dynamic lane-changing trajectory planning for autonomous vehicles based on discrete global trajectory. IEEE Trans Intell Transp Syst2022; 23(7): 8513–8527.
50.
FujimotoSvan HoofHMegerD. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:180209477, 2018.
51.
WegenerMKochLEisenbarthM, et al. Automated ecodriving in urban scenarios using deep reinforcement learning. Trans Res Part C Emerg Technol2021; 126: 102967.
52.
DesaiMGhaffariA. CLF-CBF based quadratic programs for safe motion control of nonholonomic mobile robots in presence of moving obstacles. In: 2022 IEEE/ASME international conference on advanced intelligent mechatronics (AIM), Sapporo, Japan, 11–15 July 2022, pp.16–21. New York: IEEE.
53.
LiYLvQZhuH, et al. Variable time headway policy based platoon control for heterogeneous connected vehicles with external disturbances. IEEE Trans Intell Transp Syst2022; 23(11): 21190–21200.
54.
ZhouYWangMAhnS. Distributed model predictive control approach for cooperative car-following with guaranteed local and string stability. Transp Res Part B Methodol2019; 128: 69–86.
55.
YangJChuDWangR, et al. Coupling effect modeling of driver vehicle environment factors influencing speed selections in curves. Proc IMechE, Part D: J Automobile Engineering2020; 234(7): 2066–2078.
56.
MilanésVShladoverSE. Modeling cooperative and autonomous adaptive cruise control dynamic responses using experimental data. Transp Res Part C Emerg Technol2014; 48: 285–300.
