This paper investigates connected cruise control (CCC) based on the deep reinforcement learning algorithm to mitigate oscillations and improve traffic safety in stop-and-go waves. A proximal policy optimization (PPO)-based control strategy is proposed for connected and autonomous vehicles (CAVs) in a heterogeneous longitudinal car-following platoon. The method receives the velocity and position signals of n human-driven vehicles ahead through wireless vehicle-to-vehicle communication to obtain an appropriate driving behavior and improve traffic efficiency and safety in real-time. The effectiveness and advantage of the proposed strategy are verified by using traffic simulation software (Simulation of Urban Mobility) for two simulation scenarios of trajectory curves with noticeable acceleration changes of the leading vehicle and two sections of actual traffic speed data. The results show that the proposed PPO-CCC method can successfully suppress the speed oscillations and improve the travel efficiency of CAVs in the car platoon.
DavisL. C.Effect of Adaptive Cruise Control Systems on Traffic Flow. Physical Review. E, Statistical, Nonlinear, and Soft Matter Physics, Vol. 69, No. 6, 2004, p. 066110.
3.
ChengX.YangL.ShenX.D2D for Intelligent Transportation Systems: A Feasibility Study. IEEE Transactions on Intelligent Transportation Systems, Vol. 16, No. 4, 2015, pp. 1784–1793.
4.
MilanésV.ShladoverS. E.SpringJ.NowakowskiC.KawazoeH.NakamuraM.Cooperative Adaptive Cruise Control in Real Traffic Situations. IEEE Transactions on Intelligent Transportation Systems, Vol. 15, No. 1, 2014, pp. 296–305.
5.
GerritJ. L. N.VugtsR. P. A.PloegJ.van de MolengraftM. J. G.SteinbuchM.String-Stable CACC Design and Experimental Validation: A Frequency-Domain Approach. IEEE Transactions on Vehicular Technology, Vol. 59, No. 9, 2010, pp. 4268–4279.
6.
PloegJ.vanN.Wouw de NijmeijerH.Lp String Stability of Cascaded Systems: Application to Vehicle Platooning. IEEE Transactions on Control Systems Technology, Vol. 22, No. 2, 2014, pp. 786–793.
7.
di BernardoM.SalviA.SantiniS.Distributed Consensus Strategy for Platooning of Vehicles in the Presence of Time-Varying Heterogeneous Communication Delays. IEEE Transactions on Intelligent Transportation Systems, Vol. 16, No. 1, 2015, pp. 102–112.
8.
ZhouY.AhnS.ChitturiM.NoyceD. A.Rolling Horizon Stochastic Optimal Control Strategy for ACC and CACC Under Uncertainty. Transportation Research Part C: Emerging Technologies, Vol. 83, 2017, pp. 61–76.
9.
PloegJ.ShuklaD. P.van de WouwN.NijmeijerH.Controller Synthesis for String Stability of Vehicle Platoons. IEEE Transactions on Intelligent Transportation Systems, Vol. 15, No. 2, 2014, pp. 854–865.
10.
OroszG.Connected Cruise Control: Modelling, Delay Effects, and Nonlinear Behaviour. Vehicle System Dynamics, Vol. 54, No. 8, 2016, pp. 1147–1176.
11.
ZhangL.OroszG.Motif-Based Design for Connected Vehicle Systems in Presence of Heterogeneous Connectivity Structures and Time Delays. IEEE Transactions on Intelligent Transportation Systems, Vol. 17, No. 6, 2016, pp. 1638–1651.
12.
GeJ. I.OroszG.Optimal Control of Connected Vehicle Systems. Proc., 53rd IEEE Conference on Decision and Control, Los Angeles, CA, IEEE, New York, 2014, pp. 4107–4112.
13.
GeJ. I.OroszG.Optimal Control of Connected Vehicle Systems With Communication Delay and Driver Reaction Time. IEEE Transactions on Intelligent Transportation Systems, Vol. 18, No. 8, 2017, pp. 2056–2070.
14.
WangZ.GaoY.FangC.LiuL.GuoS.LiP.Optimal Connected Cruise Control With Arbitrary Communication Delays. IEEE Systems Journal, Vol. 14, No. 2, 2020, pp. 2913–2924.
15.
GeJ. I.OroszG.Data-Driven Parameter Estimation for Optimal Connected Cruise Control. Proc., IEEE 56th Annual Conference on Decision and Control, Melbourne, VIC, Australia, IEEE, New York, 2017, pp. 3739–3744.
16.
GaoW.JiangZ. P.OzbayK.Data-Driven Adaptive Optimal Control of Connected Vehicles. IEEE Transactions on Intelligent Transportation Systems, Vol. 18, No. 5, 2017, pp. 1122–1133.
17.
LiL.LvY.WangF.Traffic Signal Timing via Deep Reinforcement Learning. IEEE/CAA Journal of Automatica Sinica, Vol. 3, No. 3, 2016, pp. 247–254.
18.
LiangX.DuX.WangG.HanZ.A Deep Reinforcement Learning Network for Traffic Light Cycle Control. IEEE Transactions on Vehicular Technology, Vol. 68, No. 2, 2019, pp. 1243–1253.
19.
WuC.KreidiehA. R.ParvateK.VinitskyE.BayenA. M.Flow: A Modular Learning Framework for Mixed Autonomy Traffic. IEEE Transactiond on Robotics, Vol. 38, No. 2, 2022, pp. 1270–1286.
20.
ZhuM.WangX.WangY.Human-Like Autonomous Car-Following Model With Deep Reinforcement Learning. Transportation Research Part C: Emerging Technologies, Vol. 97, 2018, pp. 348–368.
21.
CuiS.SeiboldB.SternR.WorkD. B.Stabilizing Traffic Flow via a Single Autonomous Vehicle: Possibilities and Limitations. Proc., IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, IEEE, New York, 2017, pp. 1336–1341.
22.
LiM.LiZ.XuC.LiuT.Deep Reinforcement Learning-Based Vehicle Driving Strategy to Reduce Crash Risks in Traffic Oscillations. Transportation Research Record: Journal of the Transportation Research Board, 2020. 2674: 42–54.
23.
ZhouM.YuY.QuX.Development of an Efficient Driving Strategy for Connected and Automated Vehicles at Signalized Intersections: A Reinforcement Learning Approach. IEEE Transactions on Intelligent Transportation Systems, Vol. 21, No. 1, 2020, pp. 433–443.
24.
QuX.YuY.ZhouM.LinC.WangX.Jointly Dampening Traffic Oscillations and Improving Energy Consumption With Electric, Connected and Automated Vehicles: A Reinforcement Learning Based Approach. Applied Energy, Vol. 257, 2020, p. 114030.
25.
ChuT.KalabićU.Model-Based Deep Reinforcement Learning for CACC in Mixed-Autonomy Vehicle Platoon. Proc., IEEE 58th Conference on Decision and Control, Nice, France, IEEE, New York, 2019, pp. 4079–4084.
26.
DesjardinsC.Chaib-DraaB.Cooperative Adaptive Cruise Control: A Reinforcement Learning Approach. IEEE Transactions on Intelligent Transportation Systems, Vol. 12, No. 4, 2011, pp. 1248–1260.
SuttonR. S.BartoA. G.Reinforcement Learning: An Introduction. The MIT Press, Cambridge, 1998.
29.
MinihV.KavukcuogluK.SilverD.RusuA. A.VenessJ.BellemareM. G.GravesA., et al. Human-level Control Through Deep Reinforcement Learning. Nature, Vol. 518, 2015, pp. 529–533.
30.
WatkinsC. J. C. H.DayanP.Q-Learning. Machine Learning, Vol. 8, 1992, pp. 279–292.
31.
MinihV.KavukcuogluK.SilverD.GravesA.AntonoglouI.WierstraD.RiedmillerM.Playing Atari With Deep Reinforcement Learning. NIPS Deep Learning Workshop. arXiv Preprint arXiv:1312.5602, 2013.
32.
WilliamsR. J.Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning, Vol. 8, 1992, pp. 229–256.
SuttonR. S.McAllesterD.SinghS.MansourY.Policy Gradient Methods for Reinforcement Learning With Function Approximation. Advances in Neural Information Processing Systems, Vol. 12, 2000, pp. 1057–1063.
35.
OroszG.WilsonR. E.StépánG.Traffic Jams: Dynamics and Control. Philosophical Transactions. Series A, Mathematical, Physical, and Engineering Sciences, Vol. 368, No. 1928, 2010, pp. 4455–4479.
36.
NagelK.WagnerP.WoeslerR.Still Flowing: Approaches to Traffic Flow and Traffic Jam Modeling. Operations Research, Vol. 51, No. 5, 2003, pp. 681–710.
37.
TreiberM.HenneckeA.HelbingD.Congested Traffic States in Empirical Observations and Microscopic Simulations. Physical Review E, Vol. 62, No. 2, 2000, pp. 1805–1824.
38.
BandoM.HasebeK.NakayamaA.ShibataA.SugiyamaY.Dynamical Model of Traffic Congestion and Numerical Simulation. Physical Review E, Vol. 51, No. 2, 1995, pp. 1035–1042.
39.
KestingA.TreiberM.SchönhofM.HelbingD.Adaptive Cruise Control Design for Active Congestion Avoidance. Transportation Research Part C: Emerging Technologies, Vol. 16, No. 6, 2008, pp. 668–683.
40.
ZhuM.WangX.TarkoA.FangS.Modeling Car-Following Behavior on Urban Expressways in Shanghai: A Naturalistic Driving Study. Transportation Research Part C: Emerging Technologies, Vol. 93, 2018, pp. 425–445.
GirshickR.DonahueJ.DarrellT.MalikJ.Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. Proc., IEEE Conference on Computer Vision and Pattern Recognition, IEEE, New York, 2014, pp. 580–587.
43.
DeweyD.Reinforcement Learning and the Reward Engineering Principle. Proc., AAAI Spring Symposia, Palo Alto, CA, 2014.
44.
BalasV. E.BalasM. M.Driver Assisting by Inverse Time to Collision. Proc., 2006 World Automation Congress, Budapest, Hungary, IEEE, New York, 2006, pp. 1–6.
45.
KestingA.TreiberM.Calibrating Car-Following Models by Using Trajectory Data: Methodological Study. Transportation Research Record: Journal of the Transportation Research Board, 2008. 2088: 148–156.
46.
XuF.HirokivT.JunichiK.HuX.LiS.ShenT.LarsE.CarlosG.Real-Time Energy Optimization of HEVs Under-Connected Environment: A Benchmark Problem and Receding Horizon-Based Solution. Control Theory and Technology, Vol. 20, 2022, pp. 145–160.
47.
XuZ.JiaoX.Robust Control of Connected Cruise Vehicle Platoon With Uncertain Human Driving Reaction Time. IEEE Transactions on Intelligent Vehicles, Vol. 7, No. 2, 2022, pp. 368–376. https://doi.org/10.1109/TIV.2021.3131641.
48.
WangZ.JinS.LiuL.FangC.LiM.GuoS.Design of Intelligent Connected Cruise Control With Vehicle-to-Vehicle Communication Delays. IEEE Transactions on Vehicular Technology, Vol. 71, No. 8, 2022, pp. 9011–9025. https://doi.org/10.1109/TVT.2022.3177008.
49.
KuuttiS.BowdenR.JoshiH.de TempleR.FallahS.End-to-End Reinforcement Learning for Autonomous Longitudinal Control Using Advantage Actor Critic With Temporal Context. Proc., IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand, IEEE, New York, 2019, pp. 2456–2462. https://doi.org/10.1109/ITSC.2019.8917387.
