Research on Dueling Double Deep Q Network Algorithm Based on Single-Step Momentum Update

Abstract

Vehicle behavior decision control plays a crucial role in the development of autonomous driving. However, existing autonomous driving behavior decision control algorithms based on deep reinforcement learning face several challenges, such as low efficiency in updating target network data and a lack of effective balancing between old and new experiences. To address these issues, this paper proposes a dueling double deep Q network (dueling DDQN) algorithm based on a single-step momentum update mechanism. Firstly, a single-step momentum update mechanism is designed to significantly improve the update speed of target network parameters and achieve a balanced weighting of old and new experiences during the parameter update process. Subsequently, the network structures of dueling networks and DDQNs are integrated to enhance the understanding capability of autonomous vehicles concerning their current states. Finally, tests are conducted on the OpenAI Gym simulation platform to validate the effectiveness of the proposed algorithm. The results verified that the dueling DDQN algorithm with single-step momentum updates contributes to improving the convergence speed of autonomous driving car behavior decisions. Compared with the DQN and DDQN algorithms, the proposed algorithm achieved a success rate increase of 6.0 and 8.4 percentage points in the challenging three-lane highway Test Scenario 1, and a success rate increase of 16.7 and 2.9 percentage points in Test Scenario 2, respectively. These findings demonstrate a safer and more efficient performance in autonomous driving decision-making.

Keywords

operations vehicle-highway automation automated vehicles driverless self-driving

Get full access to this article

View all access options for this article.

References

Ahangar

M. N.

Ahmed

Q. Z.

Khan

F. A.

Hafeez

A Survey of Autonomous Vehicles: Enabling Communication Technologies and Challenges. Sensors, Vol. 21, No. 3, 2021, p. 706.

Hoel

C. J.

Driggs-Campbell

Wolff

Laine

Kochenderfer

M. J.

Combining Planning and Deep Reinforcement Learning in Tactical Decision Making for Autonomous Driving. IEEE Transactions on Intelligent Vehicles, Vol. 5, No. 2, 2019, pp. 294–305.

Gao

Bian

Autonomous Driving of Vehicles Based on Artificial Intelligence. Journal of Intelligent & Fuzzy Systems, Vol. 41, No. 4, 2021, pp. 4955–4964.

Hoel

C. J.

Wolff

Laine

Tactical Decision-Making in Autonomous Driving by Reinforcement Learning with Uncertainty Estimation. Proc., IEEE Intelligent Vehicles Symposium (IV), Las Vegas, NV, IEEE, New York, 2020, pp. 1563–1569.

González

Pérez

Milanés

Nashashibi

A Review of Motion Planning Techniques for Automated Vehicles. IEEE Transactions on Intelligent Transportation Systems, Vol. 17, No. 4, 2015, pp. 1135–1145.

Isele

Rahimi

Cosgun

Subramanian

Fujimura

Navigating Occluded Intersections with Autonomous Vehicles Using Deep Reinforcement Learning. Proc., IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia, IEEE, New York, 2018, pp. 2034–2039.

Talebpour

Mahmassani

H. S.

Hamdar

S. H.

Modeling Lane-Changing Behavior in a Connected Environment: A Game Theory Approach. Transportation Research Procedia, Vol. 7, 2015, pp. 420–440.

Bojarski

Del Testa

Dworakowski

Firner

Flepp

Goyal

Jackel

L. D.

, et al. 2016. End to End Learning for Self-Driving Cars. arXiv Preprint arXiv:1604.07316, 2016.

Hoel

C. J.

Wolff

Laine

Automated Speed and Lane Change Decision Making Using Deep Reinforcement Learning. Proc., 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, IEEE, New York, 2018, pp. 2148–2155.

10.

Alizadeh

Moghadam

Bicer

Ure

N. K.

Yavas

Kurtulus

Automated Lane Change Decision Making Using Deep Reinforcement Learning in Dynamic and Uncertain Highway Environment. Proc., IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand, IEEE, New York, 2019, pp. 1399–1404.

11.

Shi

Wang

Cheng

Chan

C. Y.

Huang

Driving Decision and Control for Automated Lane Change Behavior Based on Deep Reinforcement Learning. Proc., IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand, IEEE, New York, 2019, pp. 2895–2900.

12.

Zuo

Qian

Ren

Sun

A Reinforcement Learning Approach to Autonomous Decision Making of Intelligent Vehicles on Highways. IEEE Transactions on Systems, Man, and Cybernetics: Systems, Vol. 50, No. 10, 2018, pp. 3884–3897.

13.

Liao

Liu

Tang

Huang

Cao

Decision-Making Strategy on Highway for Autonomous Vehicles Using Deep Reinforcement Learning. IEEE Access, Vol. 8, 2020, pp. 177804–177814.

14.

Zhao

A Deep Reinforcement Learning Approach for Autonomous Highway Driving. IFAC-PapersOnLine, Vol. 53, No. 5, 2020, pp. 542–546.

15.

Liu

Deng

Tang

Wang

Cao

Predictive Freeway Overtaking Strategy for Automated Vehicles Using Deep Reinforcement Learning. Proc., 3rd Conference on Vehicle Control and Intelligence (CVCI), Hefei, China, IEEE, New York, 2019, pp. 1–6.

16.

Puterman

M. L.

Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley Sons, Hoboken, NJ, 2014.

17.

Watkins

C. J.

Dayan

Q-learning. Machine Learning, Vol. 8, 1992, pp. 279–292.

18.

Mnih

Kavukcuoglu

Silver

Graves

Antonoglou

Wierstra

Riedmiller

Playing Atari with Deep Reinforcement Learning. arXiv Preprint arXiv:1312.5602, 2013.

19.

Wang

Schaul

Hessel

Hasselt

Lanctot

Freitas

Dueling Network Architectures for Deep Reinforcement Learning. Proceedings of the International Conference on Machine Learning, PMLR, Vol. 48, 2016, pp. 1995–2003.

20.

Van Hasselt

Guez

Silver

Deep Reinforcement Learning with Double q-Learning. Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 30, No. 1, 2016, pp. 2094–2100.

21.

Treiber

Hennecke

Helbing

Congested Traffic States in Empirical Observations and Microscopic Simulations. Physical Review E, Vol. 62, No. 2, 2000, p. 1805.

22.

Kesting

Treiber

Helbing

General Lane-Changing Model MOBIL for Car-Following Models. Transportation Research Record: Journal of the Transportation Research Board, 2007. 1999: 86–94.

23.

Yavas

Kumbasar

Ure

N. K.

A New Approach for Tactical Decision Making in Lane Changing: Sample Efficient Deep Q Learning with a Safety Feedback Reward. Proc., IEEE Intelligent Vehicles Symposium (IV), Las Vegas, NV, IEEE, New York, pp. 1156–1161.

24.

Leurent

An Environment for Autonomous Driving Decision-Making. GitHub Repository. 2018. https://github.com/Farama-Foundation/HighwayEnv