Resource-Efficient Dynamic Platooning Control through Deep Q-Network-Based Event-Triggered Scheduling

Abstract

In modern transportation systems, dynamic platooning is a promising approach, leveraging adaptable vehicle formations to optimize traffic flow and fuel efficiency. However, relying on periodic broadcasts to preserve string stability during platoon maneuvers poses a challenge, demanding substantial resources within a constrained spectrum. This paper presents an event-triggered message scheduling that uses deep reinforcement learning specifically tailored for a dynamic platoon of autonomous vehicles on highways. In the proposed approach, the platoon leader intelligently triggers communication instances only when platoon speed variations threaten string stability. Furthermore, this study introduces a categorization system for triggered messages, distinguishing between safety and non-safety messages, ensuring their efficient storage and prioritized transmission. Simulation results demonstrate the efficacy of the trained leader in maintaining string stability, even amidst joining and leaving maneuvers, achieved through reduced communication frequency. Additionally, shorter waiting time for safety messages within the priority buffer guarantees reliable and expedited communication of critical information.

Keywords

deep Q network dynamic platoon event-triggered scheduling platoon string stability resource efficiency

Get full access to this article

View all access options for this article.

References

Lesch

Breitbach

Segata

Becker

Kounev

Krupitzer

An Overview on Approaches for Coordination of Platoons. IEEE Transactions on Intelligent Transportation Systems, Vol. 23, No. 8, Aug. 2022, pp. 10049–10065. https://doi.org/10.1109/TITS.2021.3115908.

Maiti

Winter

Kulik

Sarkar

The Impact of Flexible Platoon Formation Operations. IEEE Transactions on Intelligent Vehicles, Vol. 5, No. 2, June 2020, pp. 229–239. https://doi.org/10.1109/TIV.2019.2955898.

Jia

Wang

Zhang

Shen

A Survey on Platoon-Based Vehicular Cyber-Physical Systems. IEEE Communications Surveys and Tutorials, Vol. 18, No. 1, 2016, pp. 263–284. https://doi.org/10.1109/COMST.2015.2410831.

Agbaje

Anjum

Mitra

Oseghale

Bloom

Olufowobi

Survey of Interoperability Challenges in the Internet of Vehicles. IEEE Transactions on Intelligent Transportation Systems, Vol. 23, no. 12, Dec. 2022, pp. 22838–22861. https://doi.org/10.1109/TITS.2022.3194413.

Mushtaq

Khan

Koubaa

Boulila

A Microscopic Platoon Stability Model Using Vehicle-to-Vehicle Communication. Electronics, Vol. 11, No. 13, 2022, p. 1994. https://doi.org/10.3390/electronics11131994.

Chin

W. H.

Fan

Haines

Emerging Technologies and Research Challenges for 5G Wireless Networks. IEEE Wireless Communications, Vol. 21, No. 2, April 2014, pp. 106–112. https://doi.org/10.1109/MWC.2014.6812298.

ETSI TR 137 985 V16.0.0 (2020-07) LTE, 5G. Overall description of Radio Access Network (RAN), Aspects for Vehicle-to-Everything (V2X) Based on LTE and NR. (3GPP TR 37.985 version 16.0.0 Release 16).

Kamal

M. A.

Raza

H. W.

Alam

M. M.

Su’ud

M. M.

Bakar

A. A.

Resource Allocation Schemes for 5G Network: A Systematic Review. Sensors, Vol. 21, No. 19, 2021, p. 6588. https://doi.org/10.3390/s21196588.

Hegde

Blume

Shrivastava

Bakker

Enhanced Resource Scheduling for Platooning in 5G V2X Systems. IEEE 2nd 5G World Forum (5GWF), Dresden, 2019, pp. 108–113. https://doi.org/10.1109/5GWF.2019.8911647.

10.

Gunter

Janssen

Barbour

Stern

R. E.

Work

D. B.

Model-Based String Stability of Adaptive Cruise Control Systems Using Field Data. IEEE Transactions on Intelligent Vehicles, Vol. 5, No. 1, March 2020, pp. 90–99. https://doi.org/10.1109/TIV.2019.2955368.

11.

Khound

Will

Gronwald

Design Methodology to Derive Over-Damped String Stable Adaptive Cruise Control Systems. IEEE Transactions on Intelligent Vehicles, Vol. 7, No. 1, March 2022, pp. 32–44. https://doi.org/10.1109/TIV.2021.3066056.

12.

Feng

Song

Zhang

Robust Platoon Control in Mixed Traffic Flow Based on Tube Model Predictive Control. IEEE Transactions on Intelligent Vehicles, Vol. 6, No. 4, Dec. 2021, pp. 711–722. https://doi.org/10.1109/TIV.2021.3060626.

13.

Liang

Zhou

Liu

Zheng

Hong

Zhou

Zhao

A Dynamic Resource Allocation Model Based on SMDP and DRL Algorithm for Truck Platoon in Vehicle Network. IEEE Internet of Things Journal, Vol. 9, No. 12, 2022, pp. 10295–10305. https://doi.org/10.1109/JIOT.2021.3123811.

14.

Wen

Guo

Chen

Gao

Cooperative Adaptive Cruise Control of Vehicles Using a Resource-Efficient Communication Mechanism. IEEE Transactions on Intelligent Vehicles, Vol. 4, No. 1, March 2019, pp. 127–140. https://doi.org/10.1109/TIV.2018.2886676.

15.

Xiao

Han

Q.L.

Zhang

X. M.

Ding

Dynamic Event-Triggered Scheduling and Platooning Control Co-Design for Automated Vehicles Over Vehicular Ad-Hoc Networks. IEEE/CAA Journal of Automatica Sinica, Vol. 9, No. 1, January 2022, pp. 31–46. https://doi.org/10.1109/JAS.2021.1004060.

16.

Dong

Zhu

Jiang

Zeng

Wei

Zheng

F. C.

Leung

K. C.

Dynamic Manager Selection Assisted Resource Allocation in URLLC With Finite Block Length for 5G-V2X Platoons. IEEE Transactions on Vehicular Technology, Vol. 71, No. 11, Nov. 2022, pp. 11336–11350. https://doi.org/10.1109/TVT.2022.3190518.

17.

Cao

Yin

Resource Allocation for Vehicle Platooning in 5G NR-V2X via Deep Reinforcement Learning. IEEE International Black Sea Conference on Communications and Networking (BlackSeaCom), Bucharest, Romania, 2021, pp. 1–7. https://doi.org/10.1109/BlackSeaCom52164.2021.9527765.

18.

Farzanullah

Le-Ngoc

Platoon Leader Selection, User Association and Resource Allocation on a C-V2X Based Highway: A Reinforcement Learning Approach. ICC 2023 – IEEE International Conference on Communications, Rome, Italy, 2023, pp. 5396–5401. https://doi.org/10.1109/ICC45041.2023.10279056.

19.

H. V.

Farzanullah

Liu

Nguyen

D. H. N.

Morawski

Le-Ngoc

Multi-Agent Reinforcement Learning for Channel Assignment and Power Allocation in Platoon-Based C-V2X Systems. IEEE 95th Vehicular Technology Conference (VTC2022-Spring), Helsinki, Finland, 2022, pp. 1–5. https://doi.org/10.1109/VTC2022-Spring54318.2022.9860518

20.

Wang

Zhang

Bian

Song

Platoon Cooperation in Cellular V2X Networks for 5G and Beyond. IEEE Transactions on Wireless Communications, Vol. 18, No. 8, Aug. 2019, pp. 3919–3932. https://doi.org/10.1109/TWC.2019.2919602.

21.

Ye Li

Fred Juang

B.-H.

Deep Reinforcement Learning Based Resource Allocation for V2V Communications. IEEE Transactions on Vehicular Technology, Vol. 68, No. 4, April 2019, pp. 3163–3173. https://doi.org/10.1109/TVT.2019.2897134.

22.

Sroka

Ström

Svensson

Kliks

Autonomous Controller-Aware Scheduling of Intra-Platoon V2V Communications. Sensors, Vol. 23, 2022, p. 60. https://doi.org/10.3390/s23010060.

23.

Wen

Guo

Control of Leader-Following Vehicle Platoons with Varied Communication Range. IEEE Transactions on Intelligent Vehicles, Vol. 5, No. 2, June 2020, pp. 240–250. https://doi.org/10.1109/TIV.2019.2955899.

24.

Hong

Shan

Song

Zhuang

Xiang

A Joint Design of Platoon Communication and Control Based on LTE-V2V. IEEE Transactions on Vehicular Technology, Vol. 69, No. 12, Dec. 2020, pp. 15893–15907. https://doi.org/10.1109/TVT.2020.3037239.

25.

Xiao

Han

Q. L.

Zhang

Resource-Efficient Platooning Control of Connected Automated Vehicles Over VANETs. IEEE Transactions on Intelligent Vehicles, Vol. 7, No. 3, Sept. 2022, pp. 579–589. https://doi.org/10.1109/TIV.2022.3155640.

26.

Sybis

Kryszkiewicz

Sroka

On the Context-Aware, Dynamic Spectrum Access for Robust Intraplatoon Communications. Mobile Information Systems, Vol. 2018, 2018, Article ID 3483298, 16 pages. https://doi.org/10.1155/2018/3483298.

27.

Guo

Wen

Communication Scheduling and Control of a Platoon of Vehicles in VANETs. IEEE Transactions on Intelligent Transportation Systems, Vol. 17, No. 6, June 2016, pp. 1551–1563. https://doi.org/10.1109/TITS.2015.2505407.

28.

Cao

Roy

Yin

Resource Allocation in 5G Platoon Communication: Modeling, Analysis and Optimization. IEEE Transactions on Vehicular Technology, Vol. 72, No. 4, April 2023, pp. 5035–5048. https://doi.org/10.1109/TVT.2022.3223351.

29.

Wang

Zhang

Bian

Song

Joint Platoon Formation and Resource Allocation for Connected Vehicles by Cellular V2X Communication. ICC 2019 – 2019 IEEE International Conference on Communications (ICC), Shanghai, China, 2019, pp. 1–6. https://doi.org/10.1109/ICC.2019.8761878.

30.

Jamalipour

Zhang

Federated Multi-Agent Deep Reinforcement Learning for Resource Allocation of Vehicle-to-Vehicle Communications. IEEE Transactions on Vehicular Technology, Vol. 71, No. 8, Aug. 2022, pp. 8810–8824. https://doi.org/10.1109/TVT.2022.3173057.

31.

Hongyu

Zixuan

Ming

Zhengyi

Bin

Guoying

Distributed Control of a Vehicular Platoon Using Event-Triggered Communication Strategy Based on State Estimation. Journal of Transportation Engineering, Part A: Systems, Vol. 149, No. 9, 2023, p. 04023092. https://doi.org/10.1061/JTEPBS.TEENG-7756.

32.

Wang

Zhao

Zhang

Han

Deep Q-Network-Enabled Platoon Merging Approach for Autonomous Vehicles. Transportation Research Record, 2023. 2678(7): 17–31. https://doi.org/10.1177/03611981231203229.

33.

Jiao

Proximal Policy Optimization-Based Driving Control Strategy of Connected Cruise Vehicle Platoons to Improve Traffic Efficiency and Safety. Transportation Research Record. 2023. 2677(6): 58–72. https://doi.org/10.1177/03611981221144283.

34.

Wang

Zhao

Zhang

Han

Deep Q-Network-Based Efficient Driving Strategy for Mixed Traffic Flow with Connected and Autonomous Vehicles on Urban Expressways. Transportation Research Record. 2023. 2677(10): 324–338. https://doi.org/10.1177/03611981231161355.

35.

Liu

Deep Reinforcement Learning-Based Vehicle Driving Strategy to Reduce Crash Risks in Traffic Oscillations. Transportation Research Record. 2020. 2674(10): 42–54. https://doi.org/10.1177/0361198120937976.

36.

Ammourah

Talebpour

Deep Reinforcement Learning Approach for Automated Vehicle Mandatory Lane Changing. Transportation Research Record. 2022. 2677(2): 712–724. https://doi.org/10.1177/03611981221108377.

37.

ElSamadisy

Shi

Smirnov

Abdulhai

Safe, Efficient, and Comfortable Reinforcement-Learning-Based Car-Following for AVs with an Analytic Safety Guarantee and Dynamic Target Speed. Transportation Research Record. 2023. 2678(1): 643–661. https://doi.org/10.1177/03611981231171899.

38.

Segata

Safe and Efficient Communication Protocols for Platooning Control. PhD thesis. Institute of Computer Science, University of Innsbruck, Innsbruck, Austria, Feb. 2016.

39.

Baumann

Zhu

Martius

Trimpe

Deep Reinforcement Learning for Event-Triggered Control. 2018 IEEE Conference on Decision and Control (CDC), Miami, FL, USA, 2018, pp. 943–950. https://doi.org/10.1109/CDC.2018.8619335.