Behavioral-Adaptive Deep Q-Network for Autonomous Driving Decisions in Heavy Traffic

Abstract

Deep reinforcement learning (DRL) is confronted with the significant problem of sparse rewards for autonomous driving in heavy traffic because of the dynamic and diverse nature of the driving environment as well as the complexity of the driving task. To mitigate the impact of sparse rewards on the convergence process of DRL, this paper proposes a novel behavioral-adaptive deep Q-network (BaDQN) for autonomous driving decisions in heavy traffic. BaDQN applies the idea of task decomposition to the DRL process. To break down the complexity of the driving task and achieve shorter exploration paths, BaDQN divides the driving task into three subtasks: Lane-Changing, Posture-Adjustment, and Wheel-Holding. BaDQN uses the finite state machine (FSM) to model the collaborative relationship between different subtasks, and abstracts each subtask separately using the Markov decision process (MDP). We used the Carla simulator to conduct experiments in a specific heavy traffic scenario. Compared with previous methods, BaDQN achieves a longer safe driving distance and a higher success rate. To discuss the adaptability of BaDQN to changes in traffic density and traffic velocity, we also conducted two extended experiments, which fully demonstrated the performance stability of BaDQN.