Reinforcement learning has demonstrated its potential in the decision-making field of autonomous driving. By engaging in trial-and-error learning and interacting with the environment, we can adapt our behavioral strategies based on reward and enhance driving decisions. Nevertheless, real-world vehicle decision-making involves intricate and diverse information, and the limited state information hinders agents from making optimal decisions, which may result in catastrophic consequences like collisions. Therefore, this paper proposes a robust deep reinforcement learning method based on the Soft Actor-Critic (SAC) algorithm for highway intelligent connected vehicle ramp merging decision-making. The model represents the vehicle features of the target lane and its adjacent lanes as an environmental state space. Additionally, a hybrid action space is designed, which combines discrete lateral actions and continuous longitudinal actions. Finally, an on-ramp simulation platform is built using actual roads and SUMO to verify the feasibility of the model. Multiple sets of comparative analysis experimental results demonstrate that the proposed method outperforms others in terms of the average reward return value, robustness value, collision rate, and success rate. Moreover, the model exhibits a stable lane change success rate under various road traffic density conditions, which indicates its robustness. The code can be obtained at https://github.com/ColinFanghz/sac-on-ramp.git.
ClaussmannLRevilloudMGruyerD, et al. A review of motion planning for highway autonomous driving. IEEE Trans Intell Transp Syst2019; 21(5): 1826–1848.
2.
YeFChengXWangP, et al. Automated lane change strategy using proximal policy optimization-based deep reinforcement learning. In: 31st IEEE intelligent vehicles symposium (IV), Las Vegas, NV, USA, 19 October 2020–13 November 2020, pp.1746–1752. New York: IEEE.
3.
YeFWangPChanCY, et al. Meta reinforcement learning based lane change strategy for autonomous vehicles. In: 32nd IEEE intelligent vehicles symposium (IV), Nagoya, Japan, 11–17 July 2021, pp.223–230. New York: IEEE.
4.
KestingATreiberMHelbingD.General lane-changing model MOBIL for car-following models. Transp Res Rec2007; 1999(1): 86–94.
5.
HoelCJWahdeMWolffK. An evolutionary approach to general-purpose automated speed and lane change behavior. In: 16th IEEE international conference on machine learning and applications (ICMLA), Cancun, Mexico, 18–21 December 2017, pp.743–748. New York: IEEE.
6.
DuXYaoR.Evolutionary game mechanism of mandatory lane changing for exiting for intelligent connected bus. J Jilin Univ (Eng Technol Ed)2024; 54(1): 124–135.
7.
ShangTWuPTangB, et al. Study on lane-changing game behavior of vehicles in small spacing section between tunnel and interchange. China Saf Sci J2021; 31(10): 68–75.
8.
WangHChenWWangQ, et al. Lane change decision based on the safety state division of the adjacent lanes. J Mech Eng2023; 59(2): 233–244.
9.
JiaHLiuPZhangL, et al. Lane-changing decision model development by combining rules abstract and machine learning technique. J Mech Eng2022; 58(4): 212–221.
10.
MnihVKavukcuogluKSilverD, et al. Human-level control through deep reinforcement learning. Nature2015; 518(7540): 529–533.
11.
YanXZouZFengS, et al. Learning naturalistic driving environment with statistical realism. Nat Commun2023; 14(1): 2037.
12.
LiQPengZZhouB.Efficient learning of safe driving policy via human-AI copilot optimization. arXiv preprint arXiv:220210341, 2022.
13.
WatkinsCJCH. Learning from delayed rewards. PhD dissertation, University of Cambridge, Cambridge, England, 1989
14.
MnihVKavukcuogluKSilverD, et al. Playing Atari with deep reinforcement learning. arXiv preprint arXiv:13125602, 2013.
15.
MnihVBadiaAPMirzaM, et al. Asynchronous methods for deep reinforcement learning. In: 33rd international conference on machine learning (ICML), New York, NY, USA, 19–24 June 2016, pp.1928–1937. New York: PMLR.
16.
LillicrapTPHuntJJPritzelA, et al. Continuous control with deep reinforcement learning. arXiv preprint arXiv:150902971, 2015.
FujimotoSHoofHMegerD. Addressing function approximation error in actor-critic methods. In: 35th international conference on machine learning (ICML), Stockholm, Sweden, 10–15 July 2018, pp.1587–1596. New York: PMLR.
19.
HaarnojaTZhouAAbbeelP, et al. Soft actor-critic: off policy maximum entropy deep reinforcement learning with a stochastic actor. In: 35th international conference on machine learning (ICML), Stockholm, Sweden, 10–15 July 2018, pp.1861–1870. New York: PMLR.
20.
HafnerDLillicrapTBaJ, et al. Dream to control: learning behaviors by latent imagination. arXiv preprint arXiv:191201603, 2019.
21.
JumperJEvansRPritzelA, et al. Highly accurate protein structure prediction with AlphaFold. Nature2021; 596(7873): 583–589.
22.
LiYChoiDChungJ, et al. Competition-level code generation with AlphaCode. Science2022; 378(6624): 1092–1097.
23.
HafnerDPasukonisJBaJ, et al. Mastering diverse domains through world models. arXiv preprint arXiv: 230104104, 2023.
24.
KiranBRSobhITalpaertV, et al. Deep reinforcement learning for autonomous driving: a survey. IEEE Trans Intell Transp Syst2021; 23(6): 4909–4926.
25.
HoelCJWolffKLaineL. Automated speed and lane change decision making using deep reinforcement learning. In: 21st international conference on intelligent transportation systems (ITSC), Maui, HI, USA, 4–7 November 2018, pp.2148–2155. New York: IEEE.
26.
LiGYangYLiS, et al. Decision making of autonomous vehicles in lane change scenarios: deep reinforcement learning approaches with risk awareness. Transp Res Part C Emerg Technol2022; 134: 103452.
27.
MirchevskaBPekCWerlingM, et al. High-level decision making for safe and reasonable autonomous lane changing using reinforcement learning. In: 21st international conference on intelligent transportation systems (ITSC), Maui, HI, USA, 4–7 November 2018, pp.2156–2162. New York: IEEE.
28.
WangGHuJLiZ, et al. Harmonious lane changing via deep reinforcement learning. IEEE Trans Intell Transp Syst2021; 23(5): 4642–4650.
29.
WangJZhangQZhaoD, et al. Lane change decision making through deep reinforcement learning with rule-based constraints. In: International joint conference on neural networks (IJCNN), Budapest, Hungary, 14–19 July 2019, pp.1–6. New York: IEEE.
30.
HuegleMKalweitGMirchevskaB, et al. Dynamic input for deep reinforcement learning in autonomous driving. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), Macau, China, 3–8 November 2019, pp.7566–7573. New York: IEEE.
31.
JiangSChenJShenM.An interactive lane change decision making model with deep reinforcement learning. In: 7th international conference on control, mechatronics and automation (ICCMA), TU Delft, Netherlands, 6–8 November 2019, pp.370–376. New York: IEEE.
32.
WuJHuangWde BoerN, et al. Safe decision-making for lane-change of autonomous vehicles via human demonstration aided reinforcement learning. In: 2022 25th international conference on intelligent transportation systems (ITSC), Macau, China, 8–12 October 2022, pp.1228–1233. New York: IEEE.
33.
NageshraoSTsengHEFilevD. Autonomous highway driving using deep reinforcement learning. In: International conference on systems, man and cybernetics (SMC), 6–9 October 2019, Bari, Italy, pp.2326–2331. New York: IEEE.
34.
ZhangYSunPYinY, et al. Human-like autonomous vehicle speed control by deep reinforcement learning with double Q learning. In: 29th IEEE intelligent vehicles symposium (IV), Changshu, China, 26–30 June 2018, pp.1251–1256. New York: IEEE.
35.
FuYLiCYuFR, et al. A decision-making strategy for vehicle autonomous braking in emergency via deep reinforcement learning. IEEE Trans Intell Veh2020; 69(6): 5876–5888.
36.
YeYZhangXSunJ.Automated vehicle’s behavior decision making using deep reinforcement learning and high-fidelity simulation environment. Transp Res Part C Emerg Technol2019; 107: 155–170.
37.
ZhuMWangXWangY.Human-like autonomous car following model with deep reinforcement learning. Transp Res Part C Emerg Technol2018; 97: 348–368.
38.
HartmannGShillerZAzariaA. Deep reinforcement learning for time optimal velocity control using prior knowledge. In: 31st international conference on tools with artificial intelligence (ICTAI), Portland, OR, USA, 4–6 November 2019, pp.186–193. New York: IEEE.
39.
HeXLouBYangH, et al. Robust decision making for autonomous vehicles at highway on-ramps: a constrained adversarial reinforcement learning approach. IEEE Trans Intell Transp Syst2022; 24(4): 4103–4113.
40.
HeXLvCJiX, et al. Emergency collision avoidance decision making for autonomous vehicles: a model-based reinforcement learning approach. In: 6th CAA international conference on vehicular control and intelligence (CVCI), Nanjing, China, 28–30 October 2022, pp.1–8. New York: IEEE.
41.
HeXYangHHuZ, et al. Robust lane change decision making for autonomous vehicles: an observation adversarial reinforcement learning approach. IEEE Trans Intell Veh2022; 8(1): 184–193.
42.
ErdmannJ.Sumo’s lane-changing model. In: BehrischMWeberM (eds) Modeling mobility with open data: 2nd SUMO conference. Cham: Springer, 2015, pp.105–123.
43.
TreiberMHenneckeAHelbingD.Congested traffic states in empirical observations and microscopic simulations. Phys Rev E2000; 62(2): 1805.
