Deep reinforcement learning (DRL) based car-following control (CFC) models are widely applied in the longitudinal motion control tasks of automated vehicles by self-learning for the optimal control policy. However, DRL algorithms easily produce unsafe commands and have low robustness, especially in complex car-following scenarios. To improve the DRL-based CFC model, this paper combines the deep deterministic policy gradient (DDPG) based CFC model with the deep optical flow estimation (DOFE) based CFC model that can overcome the shortcomings of DDPG-based one which is denoted as cooperative car-following model (DDPGoF). The DDPG-based CFC model utilizes prioritized experience replay which can intrinsically accelerate the learning speed. Meanwhile, the proposed DOFE-based CFC model employs the recurrent all-pairs field transforms algorithm (RAFT) and EfficientNet to perceive the motion variation of the surrounding vehicles, motorcycles, etc. The real vehicle driving data sets are applied to calibrate and validate the proposed DDPGoF-based CFC model while several assessment criteria are established to evaluate its overall performance. As a result, the DDPGoF-based CFC model is superior to DDPG-based one in avoiding crashes, improving car-following stability, riding comfort, and fuel economy of HEV.
ChanCCChauKT. Modern electric vehicle technology. New York, NY: Oxford University Press on Demand, 2001.
2.
ZhaoDHuangXPengH, et al. Accelerated evaluation of automated vehicles in car-following maneuvers. IEEE Trans Intell Transp Syst2017; 19(3): 733–744.
3.
LiLMaW. A collision-free car-following model for connected automated vehicles. In: Transportation research board 96th annual meeting, Washington DC, 2017, pp.1–19. Washington, DC: The National Academies of Sciences, Engineering, and Medicine.
4.
JanstrupKH. Road safety annual report 2017. Lyngby, Denmark: Technical University of Denmark, 2017.
5.
VitsAHodac–AceaIMosséO, et al. Final report of the esafety working group on road safety. European Commission, November2002.
6.
YangDZhuLLiuY, et al. A novel car-following control model combining machine learning and kinematics models for automated vehicles. IEEE Trans Intell Transp Syst2018; 20(6): 1991–2000.
7.
ShaoutAJarrahMA. Cruise control technology review. Comput Electr Eng1997; 23(4): 259–271.
8.
GanjiBKouzaniAZKhooSY, et al. Adaptive cruise control of a HEV using sliding mode control. Expert Syst Appl2014; 41(2): 607–615.
9.
BleekR. Design of a hybrid adaptive cruise control stop-&-go system[J]. Technische Universiteit Eindhoven, 2007.
10.
ZhuLGonderJBjarkvikE, et al. An automated vehicle fuel economy benefits evaluation framework using real-world travel and traffic data. IEEE Trans Intell Transp Syst2019; 11(3): 29–41.
11.
JanaiJGüneyFBehlA, et al. Computer vision for autonomous vehicles: problems, datasets and state of the art. Found Trends Comput Graph Vis2020; 12(1–3): 1–308.
12.
TeedZDengJ. Raft: recurrent all-pairs field transforms for optical flow. In: Computer vision–ECCV 2020: 16th European conference, Proceedings, Part II 16, Glasgow, 23–28 August 2020, pp.402–419. Cham: Springer International Publishing.
13.
SunLPengCZhanW, et al. A fast integrated planning and control framework for autonomous driving via imitation learning. In: Dynamic systems and control conference, No. V003T37A012, Atlanta, GA, USA, 30 September–3 October 2018. New York: American Society of Mechanical Engineers.
14.
CodevillaFMüllerMLópezA, et al. End-to-end driving via conditional imitation learning. In: IEEE international conference on robotics and automation (ICRA), Brisbane, QLD, Australia, 2018, pp.4693–4700. New York: IEEE.
15.
SamakTVSamakCVKandhasamyS. Robust behavioral cloning for autonomous vehicles using end-to-end imitation learning. arXiv preprint arXiv:2010.04767, 2020.
16.
CodevillaFSantanaELópezA M, et al. Exploring the limitations of behavior cloning for autonomous driving. In: Proceedings of the IEEE/CVF international conference on computer vision, Long Beach, CA, USA, 2019, pp.9329–9338. New York: IEEE.
17.
FrijiHGhazzaiHBesbesH, et al. A DQN-based autonomous car-following framework using RGB-D frames. In: IEEE global conference on artificial intelligence and internet of things (GCAIoT), Dubai, United Arab Emirates, 12–16 December 2020, pp.1–6. New York: IEEE.
18.
LinYMcPheeJAzadNL. Longitudinal dynamic versus kinematic models for car-following control using deep reinforcement learning. In: IEEE intelligent transportation systems conference (ITSC), Auckland, New Zealand, 2019, pp.1504–1510. New York: IEEE.
19.
LinYMcPheeJAzadNL. Comparison of deep reinforcement learning and model predictive control for adaptive cruise control. IEEE Trans Intell Veh2020; 6(2): 221–231.
20.
LiYHeHPengJ, et al. Deep reinforcement learning-based energy management for a series hybrid electric vehicle enabled by history cumulative trip information. IEEE Trans Veh Technol2019; 68(8): 7416–7430.
21.
LianRPengJWuY, et al. Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle. Energy2020; 197: 117297.
22.
TanHZhangHPengJ, et al. Energy management of hybrid electric bus based on deep reinforcement learning in continuous state and action space. Energy Convers Manag2019; 195: 548–560.
23.
WuYTanHPengJ, et al. Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus. Appl Energy2019; 247: 454–466.
24.
WangZSchaulTHesselM, et al. Dueling network architectures for deep reinforcement learning. In: Proceedings of the 33rd international conference on machine learning, New York, 2016, pp.1995–2003. PMLR.
25.
RajamaniR. Vehicle dynamics and control. New York, NY: Springer Science & Business Media, 2011.
26.
SantanaEHotzG. Learning a driving simulator. arXiv preprint arXiv:1608.01230, 2016.
27.
TanMLeQ. Efficientnet: rethinking model scaling for convolutional neural networks. In: Proceedings of the 36th international conference on machine learning, Long Beach, CA, USA, 2019, pp.6105–6114. PMLR.
28.
HornBKSchunckBG. Determining optical flow. In: PearsonJJ (ed.) Techniques and applications of image understanding. Vol. 281. Bellingham, WA: International Society for Optics and Photonics, 1981, pp.319–331.
29.
ZachCPockTBischofH. A duality based approach for realtime tv-l1 optical flow. In: Pattern recognition: 29th DAGM symposium, Proceedings 29, Heidelberg, Germany, 12–14September2007, pp.214–223. Berlin, Heidelberg: Springer.
30.
ChenQKoltunV. Full flow: optical flow estimation by global optimization over regular grids. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 2016, pp.4706–4714. New York: IEEE.
31.
RaghuMPooleBKleinbergJ, et al. On the expressive power of deep neural networks. In: Proceedings of the 34th international conference on machine learning, Sydney, NSW, Australia, 2017, pp.847–2854. PMLR.
32.
LuZPuHWangF, et al. The expressive power of neural networks: a view from the width. In: GuyonIVon LuxburgUBengioS, et al. (eds) Advances in neural information processing systems, Vol. 30, 2017. Red Hook, NY: Curran Associates, Inc.
ZophBVasudevanVShlensJ, et al. Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Salt Lake City, UT, 2018, pp.8697–8710. New York: IEEE.
35.
RealEAggarwalAHuangY, et al. Regularized evolution for image classifier architecture search. Proc Conf AAAI Artif Intell2019; 33(01): 4780–4789.
36.
CaiHZhuLHanS. Proxylessnas: direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.00332, 2018.
TanMChenBPangR, et al. Mnasnet: platform-aware neural architecture search for mobile. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Long Beach, CA, USA, 2019, pp.2820–2828. New York: IEEE.
39.
IoffeSSzegedyC. Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd international conference on machine learning, Lille, France, 2015, pp.448–456. PMLR.
40.
RamachandranPZophBLeQV. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
41.
SandlerMHowardAZhuM, et al. Mobilenetv2: inverted residuals | linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Salt Lake City, UT, USA, 2018, pp.4510–4520. New York: IEEE.
42.
CholletF. Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, HI, USA, 2017, pp.1251–1258. New York: IEEE.
43.
HeKZhangXRenS, et al. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 2016, pp.770–778. New York: IEEE.
44.
ElfwingSUchibeEDoyaK. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Netw2018; 107: 3–11.
45.
LillicrapTPHuntJJPritzelA, et al. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
46.
HouYLiuLWeiQ, et al. A novel DDPG method with prioritized experience replay. In: IEEE international conference on systems, man, and cybernetics (SMC), Banff, AB, Canada, 5–8 October 2017, pp.316–321. New York: IEEE.
47.
IslamRHendersonPGomrokchiM, et al. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. arXiv preprint arXiv:1708.04133, 2017.
48.
LarochelleHBengioYLouradourJ, et al. Exploring strategies for training deep neural networks. J Mach Learn Res2009; 10(1): 1–40.
