Sage Journals: Discover world-class research

Abstract

In complex highway scenarios, autonomous driving systems require robust and adaptive decision-making capabilities to ensure safe and efficient driving operations amidst dynamic multi-vehicle interactions. To address the limitations of traditional vector data inputs in high-dimensional feature perception and the excessive redundancy in end-to-end image inputs, this paper proposes a deep reinforcement learning method based on an image-enhanced data framework SAC (IEDF–SAC). This approach introduces an innovative data representation structure that retains critical traffic object features essential for decision-making while mitigating the interference of environmental background and platform device parameters on the policy, thereby significantly enhancing the policy’s portability and generalization ability. In the feature extraction phase, a lightweight MobileNetV3-Small network is employed as the image feature encoder, incorporating inverted residual structures, attention mechanisms, and efficient activation functions. This enables rapid processing of continuous multi-frame images at a 64 × 64 resolution, achieving efficient and real-time feature extraction. The policy network, built on the soft actor-critic (SAC) algorithm, facilitates smooth control of continuous actions and is trained and validated in a self-constructed multi-vehicle dynamic interference highway simulation environment. Extensive experimental results demonstrate that IEDF–SAC outperforms comparative algorithms such as DDPG and baseline SAC across comprehensive metrics including safety, stability, and driving efficiency. Ablation studies further validate the effectiveness of the image-enhanced data framework. This research not only advances the application of continuous-action deep reinforcement learning in autonomous driving scenarios but also provides a scalable technical solution for the real-time deployment of intelligent transportation systems.

Keywords

autonomous driving highway-scenarios MobileNetV3-Small IEDF SAC

Get full access to this article

View all access options for this article.

References

Wang

Yang

, et al. Artificial intelligence applications in the development of autonomous vehicles: a survey. IEEE/CAA J Autom Sin 2020; 7: 315–329.

Aradi

SAR

. Survey of deep reinforcement learning for motion planning of autonomous vehicles. IEEE Trans Intell Transp Syst 2020; 23: 740–759.

Kiran

Sobh

Talpaert

, et al. Deep reinforcement learning for autonomous driving: a survey. IEEE Trans Intell Transp Syst 2021; 23: 4909–4926.

Hoel

Driggs-Campbell

Wolff

, et al. Combining planning and deep reinforcement learning in tactical decision making for autonomous driving. IEEE Trans Intell Veh 2019; 5: 294–305.

Alizadeh

Moghadam

Bicer

, et al. Automated lane change decision making using deep reinforcement learning in dynamic and uncertain highway environment. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand, 2019. pp. 1399–1404. Piscataway: IEEE.

Liao

Liu

, et al. Deep reinforcement learning on autonomous driving policy with auxiliary critic network. IEEE Trans Neural Netw Learn Syst 2021; 34: 3680–3690.

Mnih

Kavukcuoglu

Silver

, et al. Playing Atari with deep reinforcement learning. arXiv preprint arXiv: 1312.5602, 2013.

Schulman

Wolski

Dhariwal

, et al. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.

Wang

Chan

. Continuous control for automated lane change behavior based on deep deterministic policy gradient algorithm. In: 2019 IEEE Intelligent Vehicles Symposium (IV), Paris, France, 2019, pp. 1454–1460. Piscataway: IEEE.

10.

Yuan

Yang

, et al. Multi-reward architecture based reinforcement learning for highway driving policies. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand, 2019, pp. 3810–3815. Piscataway: IEEE.

11.

Wang

Huang

Tang

, et al. A deep reinforcement learning-based approach for autonomous lane-changing velocity control in mixed flow of vehicle group level. Expert Syst Appl 2024; 238: 122158.

12.

Lillicrap

Hunt

Pritzel

, et al. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.

13.

Haarnoja

Zhou

Abbeel

, et al. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden, 10–15 July 2018, Vol. 80. Proceedings of Machine Learning Research, pp. 1861–1870. Cambridge: PMLR.

14.

Fujimoto

Hoof

Meger

. Addressing function approximation error in actor-critic methods. In: Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden, 10–15 July 2018, Vol. 80. Proceedings of Machine Learning Research, pp. 1587–1596. Cambridge: PMLR.

15.

Rizehvandi

Azadi

. Highway decision-making strategy for autonomous vehicle for overtaking maneuver using deep reinforcement learning (DRL) method. Amirkabir J Mech Eng 2024; 56: 595–620.

16.

Chen

Dong

Palanisamy

, et al. Attention-based hierarchical deep reinforcement learning for lane change behaviors in autonomous driving. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA, 2019, pp. 1326–1334. Piscataway: IEEE.

17.

Dong

Chen

, et al. Spatio-weighted information fusion and DRL-based control for connected autonomous vehicles. In: 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), Rhodes, Greece, 2020, pp. 1–6. Piscataway: IEEE.

18.

Dosovitskiy

Ros

Codevilla

, et al. CARLA: an open urban driving simulator. In: Proceedings of the 1st Annual Conference on Robot Learning, 13–15 November 2017. Vol. 78. Proceedings of Machine Learning Research, 2017, pp. 1–16. Cambridge: PMLR.

19.

Davarynejad

Hegyi

Vrancken

, et al. Motorway ramp-metering control with queuing consideration using Q-learning. In: 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC), Washington, DC, 2011, pp. 1652–1658. Piscataway: IEEE.

20.

Liu

, et al. Reinforcement learning-based variable speed limit control strategy to reduce traffic congestion at freeway recurrent bottlenecks. IEEE Trans Intell Transp Syst 2017; 18: 3204–3217.

21.

Haydari

Yılmaz

. Deep reinforcement learning for intelligent transportation systems: a survey. IEEE Trans Intell Transp Syst 2020; 23: 11–32.

22.

Sun

Jamshidnejad

De Schutter

. Adaptive parameterized control for coordinated traffic management using reinforcement learning. IFAC-PapersOnLine 2023; 56: 5463–5468.

23.

Zhang

Liu

Zhou

, et al. Solving dynamic traveling salesman problems with deep reinforcement learning. IEEE Trans Neural Netw Learn Syst 2021; 34: 2119–2132.

24.

Yang

Tang

Qiu

, et al. Towards robust decision-making for autonomous driving on highway. IEEE Trans Veh Technol 2023; 72: 11251–11263.

25.

Wang

Zhang

, et al. Integrated traffic control for freeway recurrent bottleneck based on deep reinforcement learning. IEEE Trans Intell Transp Syst 2022; 23: 15522–15535.

26.

Dulac-Arnold

Levine

Mankowitz

, et al. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Mach Learn 2021; 110: 2419–2468.

27.

Howard

Sandler

Chu

, et al. Searching for MobileNetV3. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019, pp. 1314–1324. Piscataway: IEEE.

28.

Lee

Yoon

Kim

, et al. Ego vehicle speed estimation with automotive corner radar under velocity ambiguity using optimized MobileNetV3. IEEE Access 2025; 2025: 1.

29.

Leurent

. An environment for autonomous driving decision-making. 2018. https://github.com/eleurent/highway-env.

30.

Treiber

Hennecke

Helbing

. Congested traffic states in empirical observations and microscopic simulations. Phys Rev E 2000; 62: 1805.

Application of SAC algorithm with an image-enhanced data framework for autonomous driving in highway scenarios

Abstract

Keywords

Get full access to this article

References