Restricted accessResearch articleFirst published online 2025-12
Sequential game guided multi-agent reinforcement learning approach for cooperative decision making of autonomous vehicles at unsignalized intersections in mixed-autonomy traffic
The advancement of Connected and Autonomous Vehicle (CAV) technology has made the complex cooperation of multiple CAVs feasible, especially at non-signalized intersections. However, challenges remain in the mixed-autonomy traffic scenarios, where collaboration among CAVs and their interactions with human-driven vehicles (HDVs) must be resolved simultaneously. Accordingly, this study proposes a new multi-agent reinforcement learning algorithm, SG-QMIX, extending the original QMIX framework by incorporating a sequential game theory-based action filter module. This module integrates the First-Come-First-Served (FCFS) strategy with the sequential game and its derived equilibrium to mask out the unavailable actions, in which the FCFS strategy simplifies the ensuing cooperation of multiple CAVs before they enter the intersection, and the equilibrium of the designed sequential game guides the CAVs to learn the interactive strategies with other participants at the junction. Moreover, the experimental results show that the SG-QMIX approach can outperform the existing approaches in terms of episodic return, episodic length, and success rate for cooperative decision-making at an isolated signal-free intersection. Additionally, the trained policy can reduce average fuel consumption, maintain traffic efficiency, and enable CAVs to interact with surrounding HDVs rationally and altruistically. Furthermore, we also validate the zero-shot transferability of SG-QMIX to an unseen scenario and its applicability to more complex intersections.
WangZBianYShladoverSE, et al. A survey on cooperative longitudinal motion control of multiple connected and automated vehicles. IEEE Intell Transp Syst Mag2019; 12(1): 4–24.
2.
LiuWHuaMDengZ, et al. A systematic survey of control techniques and applications: From autonomous vehicles to connected and automated vehicles. IEEE Internet Things J 2023; arXiv:230305665 2023.
DingJXuHHuJ, et al. Centralized cooperative intersection control under automated vehicle environment. In: 2017 IEEE intelligent vehicles symposium (IV), pp. 972–977. IEEE.
5.
ZohdyIHRakhaHA.Intersection management via vehicle connectivity: The intersection cooperative adaptive cruise control system concept. J Intell Transp Syst2016; 20(1): 17–32.
6.
IseleDRahimiRCosgunA, et al. Navigating occluded intersections with autonomous vehicles using deep reinforcement learning. In: 2018 IEEE international conference on robotics and automation (ICRA), pp. 2034–2039. IEEE. ISBN 1538630818.
7.
GuanYRenYLiSE, et al. Centralized cooperation for connected and automated vehicles at intersections by proximal policy optimization. IEEE Trans Veh Technol2020; PP(99): 1.
8.
ShuHLiuTMuX, et al. Driving tasks transfer using deep reinforcement learning for decision-making of autonomous vehicles in unsignalized intersection. IEEE Trans Veh Technol2021; 71(1): 41–52.
9.
WangZZhuangYGuQ, et al. Reinforcement learning based negotiation-aware motion planning of autonomous vehicles. In: 2021 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 4532–4537. IEEE.
10.
LiMCaoZLiZ.A reinforcement learning-based vehicle platoon control strategy for reducing energy consumption in traffic oscillations. IEEE Trans Neural Netw Learn Syst2021; 32(12): 5309–5322.
11.
MnihVKavukcuogluKSilverD, et al. Human-level control through deep reinforcement learning. Nature2015; 518(7540): 529–533.
12.
SchulmanJWolskiFDhariwalP, et al. Proximal policy optimization algorithms. 20 July 2017. arXiv preprint arXiv:170706347.
13.
YuanMShanJMiK.Deep reinforcement learning based game-theoretic decision-making for autonomous vehicles. IEEE Robot Automat Lett2021; 7(2): 818–825.
14.
LopezVGLewisFLLiuM, et al. Game-theoretic lane-changing decision making and payoff learning for autonomous vehicles. IEEE Trans Veh Technol2022; 71(4): 3609–3620.
15.
LiuMWanYLewisFL, et al. A three-level game-theoretic decision-making framework for autonomous vehicles. IEEE Trans Intell Transp Syst2022; 23(11): 20298–20308.
16.
ChandraRManochaD.Gameplan: Game-theoretic multi-agent planning with human drivers at intersections, roundabouts, and merging. IEEE Robot Automat Lett2022; 7(2): 2676–2683.
17.
TanM. Multi-agent reinforcement learning: independent vs. cooperative agents. In: Proceedings of the tenth international conference on machine learning. pp. 330–337.
18.
Hernandez-LealPKartalBTaylorME.A survey and critique of multiagent deep reinforcement learning. Autonom Agents Multi-Agent Syst2019; 33(6): 750–797.
19.
PapoudakisGChristianosFRahmanA, et al. Dealing with non-stationarity in multi-agent deep reinforcement learning. 11 June 2019. arXiv preprint arXiv:1906.04737.
20.
RashidTSamvelyanMSchroederC, et al. Qmix: monotonic value function factorisation for deep multi-agent reinforcement learning. In International conference on machine learning, pp. 4295–4304. PMLR. ISBN 2640–3498.
21.
SunehagPLeverGGruslysA, et al. Value-decomposition networks for cooperative multi-agent learning based on team reward. In: AAMAS ’18: Proceedings of the 17th international conference on autonomous agents and multiagent systems, pp. 2085–2087.
22.
HuJJiangSHardingSA, et al. Riit: Rethinking the importance of implementation tricks in multi-agent reinforcement learning. The Second Blogpost Track at ICLR 2023. arXiv preprint arXiv:210203479 2021.
23.
CandelaEParadaLMarquesL, et al. Transferring multi-agent reinforcement learning policies for autonomous driving using sim-to-real. In: 2022 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 8814–8820. IEEE.
24.
PaullLTaniJAhnH, et al. Duckietown: an open, inexpensive and flexible platform for autonomy education and research. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 1497–1504. IEEE.
25.
CaiPWangHSunY, et al. Dq-gat: Towards safe and efficient autonomous driving with deep q-learning and graph attention networks. IEEE Trans Intell Transp Syst2022; 23(11): 21102–21112.
26.
PengBKeskinMFKulcsárB, et al. Connected autonomous vehicles for improving mixed traffic efficiency in unsignalized intersections with deep reinforcement learning. Commun Transp Res2021; 1(1): 100017–100017.
27.
VinitskyEKreidiehALe FlemL, et al. Benchmarks for reinforcement learning in mixed-autonomy traffic. In Conference on robot learning. PMLR, pp. 399–409.
28.
AntonioGPMaria-DoloresC.Multi-agent deep reinforcement learning to manage connected autonomous vehicles at tomorrow’s intersections. IEEE Trans Veh Technol2022; 71(7): 7033–7043.
29.
WuYChenHZhuF.Dcl-aim: decentralized coordination learning of autonomous intersection management for connected and automated vehicles. Transp Res Part C Emerg Technol2019; 103: 246–260.
30.
PengZLiQHuiKM, et al. Learning to simulate self-driven particles system with coordinated policy optimization. Adv Neural Inform Process Syst2021; 34: 10784–10797.
31.
NakkaSKSChalakiBMalikopoulosAA. A multi-agent deep reinforcement learning coordination framework for connected and automated vehicles at merging roadways. In: 2022 American control conference (ACC), pp. 3297–3302. IEEE.
32.
ChenDHajidavallooMRLiZ, et al. Deep multi-agent reinforcement learning for highway on-ramp merging in mixed traffic. IEEE Trans Intell Transp Syst2023; 24: 11623–11638.
33.
LiNYaoYKolmanovskyI, et al. Game-theoretic modeling of multi-vehicle interactions at uncontrolled intersections. IEEE Trans Intell Transp Syst2020; 23(2): 1428–1442.
34.
LiDLiuGXiaoB.Human-like driving decision at unsignalized intersections based on game theory. Proc Inst Mech Eng Part D J Autom Eng2023; 237(1): 159–173.
35.
HangPLvCHuangC, et al. Cooperative decision making of connected automated vehicles at multi-lane merging zone: a coalitional game approach. IEEE Trans Intell Transp Syst2021; 23(4): 3829–3841.
36.
AmatoCChowdharyGGeramifardA, et al. Decentralized control of partially observable markov decision processes. In: 52nd IEEE conference on decision and control, pp. 2398–2405. IEEE. ISBN 1467357170.
37.
SonKKimDKangWJ, et al. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In: International conference on machine learning, pp. 5887–5896. PMLR. ISBN 2640-3498.
38.
GibbonsR, et al. A primer in game theory. New York: Harvester Wheatsheaf, 1992.
39.
Leyton-BrownKShohamY.Essentials of game theory: a concise multidisciplinary introduction. Springer Nature, 2022.
40.
ChungJ.Empirical evaluation of gated recurrent neural networks on sequence modeling. 11 December 2014. arXiv preprint arXiv:1412.3555.
41.
LopezPABehrischMBieker-WalzL, et al. Microscopic traffic simulation using sumo. In: 2018 21st international conference on intelligent transportation systems (ITSC), pp. 2575–2582. IEEE.
42.
YuCVeluAVinitskyE, et al. The surprising effectiveness of PPO in cooperative multi-agent games. Adv Neural Inform Process Syst2022; 35: 24611–24624.
43.
LoweRWuYITamarA, et al. Multi-agent actor-critic for mixed cooperative-competitive environments. Adv Neural Inform Process Syst2017; 30: 6382–6393.
44.
De WittCSGuptaTMakoviichukD, et al. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533. 18 November 2020.
45.
SamvelyanMRashidTde WittCS, et al. The starcraft multi-agent challenge. In: AAMAS ’19: Proceedings of the 18th international conference on autonomous agents and multiagent systems, pp. 2186–2188.
46.
Dulac-ArnoldGLevineNMankowitzDJ, et al. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Mach Learn2021; 110(9): 2419–2468.
