This review provides a comprehensive analysis of cooperative Multi-Agent Reinforcement Learning (MARL) approaches for robotic systems, with particular emphasis on methodological foundations, practical implementations, and emerging challenges. We first examine the evolution of distributed intelligence in robotics, tracing its development from early architectures to modern learning-based frameworks. Our analysis focuses on two complementary paradigms: cooperative methods utilizing Centralized Training with Decentralized Execution (CTDE), and hierarchical approaches that address complexity through temporal and task decomposition. We systematically compare actor-critic methods, value-based approaches, and hierarchical frameworks across theoretical foundations, implementation characteristics, and application domains spanning aerial, ground, and maritime robotics. Our comparative analysis reveals important trade-offs between expressiveness, computational efficiency, and implementation complexity, highlighting that method selection must align with specific application requirements. Furthermore, we identify critical challenges, including the sim-to-real gap, scalability constraints, communication limitations, safety verification, and coordination in heterogeneous teams, mapping promising research directions to address these barriers to widespread deployment. This survey bridges theoretical understanding with practical implementation, providing a structured framework for researchers and practitioners working on multi-agent learning for advanced robotic systems.
BondAHGasserL. Chapter 1 - an analysis of problems and research in DAI. In: Readings in distributed artificial intelligence, 1988, pp.3–35. Morgan Kaufmann.
2.
DeckerKS. Distributed problem-solving techniques: a survey. IEEE Trans Syst Man Cybern1987; 17: 729–740.
3.
StonePVelosoM. Multiagent systems: a survey from a machine learning perspective. Auton Robots2000; 8: 345–383.
4.
ParkerL. Alliance: an architecture for fault tolerant multirobot cooperation. IEEE Trans Rob Autom1998; 14: 220–240.
5.
DudekGJenkinMMiliosE, et al.A taxonomy for multi-agent robotics. Auton Robots1996; 3: 375–397.
6.
BusoniuLBabuskaRDe SchutterB. A comprehensive survey of multiagent reinforcement learning. IEEE Trans Syst Man Cybern Part C Appl Rev2008; 38: 156–172.
7.
WeissG. Multiagent systems. 2nd ed. Cambridge MA: MIT Press, 2013.
8.
AlbrechtSVChristianosFSchäferL. Multi-agent reinforcement learning: foundations and modern approaches. Cambridge MA: MIT Press, 2024.
9.
ZhouYvan KampenEJChuQ. Hybrid hierarchical reinforcement learning for online guidance and navigation with partial observability. Neurocomputing2019; 331: 443–457.
10.
ZhouYHoHW. Online robot guidance and navigation in non-stationary environment with hybrid hierarchical reinforcement learning. Eng Appl Artif Intell2022; 114: 105152.
11.
PhamHXLaHMFeil-SeiferD, et al.A distributed control framework of multiple unmanned aerial vehicles for dynamic wildfire tracking. IEEE Trans Syst Man Cybern Syst2020; 50: 1537–1548.
12.
KaufmannEBauersfeldLLoquercioA, et al.Champion-level drone racing using deep reinforcement learning. Nature2023; 620: 982–987.
13.
KongWZhouDDuY, et al.Hierarchical multi-agent reinforcement learning for multi-aircraft close-range air combat. IET Control Theory Appl2023; 17: 1840–1862.
14.
ShermanPBezzoN. A heterogeneous system of systems framework for proactive path planning of a UAV-assisted UGV in uncertain environments. In: 2024 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp.13237–13244. IEEE.
15.
KrnjaicASteleacRDThomasJD, et al.Scalable multi-agent reinforcement learning for warehouse logistics with robotic and human co-workers. In: 2024 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp.677–684. IEEE.
16.
NguyenTTNguyenNDNahavandiS. Deep reinforcement learning for multiagent systems: a review of challenges, solutions, and applications. IEEE Trans Cybern2020; 50: 3826–3839.
17.
Hernandez-LealPKartalBTaylorME. A survey and critique of multiagent deep reinforcement learning. Auton Agent Multi Agent Syst2019; 33: 750–797.
18.
EllisBCookJMoallaS, et al.SMACv2: an improved benchmark for cooperative multi-agent reinforcement learning. Adv Neural Inf Process Syst2023; 36: 37567–37593.
19.
SuttonRSBartoAG. Reinforcement learning: an introduction. Cambridge, MA, USA: MIT Press, 2018.
20.
LoweRWuYTamarA, et al.Multi-agent actor-critic for mixed cooperative-competitive environments. In: Proceedings of the 31st international conference on neural information processing systems. NIPS’17, Red Hook, NY, USA, pp.6382–6393.
21.
ZhangKYangZBaşarT. Multi-agent reinforcement learning: a selective overview of theories and algorithms. Handb Reinf Learn Control2021; 18: 321–384.
22.
NachumOTangHLuX, et al.Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint arXiv:190910618, 2019.
23.
HuGLiHLiuS, et al.Neuronsmae: a novel multi-agent reinforcement learning environment for cooperative and competitive multi-robot tasks. In: 2023 International joint conference on neural networks (IJCNN), pp.1–8.
24.
ZhangSQZhangQLinJ. Efficient communication in multi-agent reinforcement learning via variance based control. Adv Neural Inf Process Syst2019; 32: 3212–3221.
25.
LittmanML. Markov games as a framework for multi-agent reinforcement learning. In: Machine learning proceedings 1994, 1994, pp.157–163. Elsevier.
26.
MalyshevaASungTTSohnCB, et al.Deep multi-agent reinforcement learning with relevance graphs. arXiv preprint arXiv:181112557, 2018.
27.
OliehoekFAAmatoCOliehoekFA, et al.The decentralized POMDP framework. A concise introduction to decentralized POMDPs, 2016, pp.11–32.
28.
AmatoC. A first introduction to cooperative multi-agent reinforcement learning. arXiv preprint arXiv:240506161, 2024.
29.
DuWDingS. A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications. Artif Intell Rev2021; 54: 3215–3238.
30.
AmatoC. An introduction to centralized training for decentralized execution in cooperative multi-agent reinforcement learning. arXiv preprint arXiv:240903052, 2024.
31.
BernsteinDSGivanRImmermanN, et al.The complexity of decentralized control of Markov decision processes. Math Oper Res2002; 27: 819–840.
32.
RashidTSamvelyanMDe WittCS, et al.Monotonic value function factorisation for deep multi-agent reinforcement learning. J Mach Learn Res2020; 21: 1–51.
33.
YuCVeluAVinitskyE, et al.The surprising effectiveness of PPO in cooperative multi-agent games. Adv Neural Inf Process Syst2022; 35: 24611–24624.
34.
LillicrapTPHuntJJPritzelA, et al.Continuous control with deep reinforcement learning. arXiv preprint arXiv:150902971, 2015.
35.
ZhangYMouZGaoF, et al.UAV-enabled secure communications by multi-agent deep reinforcement learning. IEEE Trans Veh Technol2020; 69: 11599–11611.
36.
ZhangRZongQZhangX, et al.Game of drones: multi-UAV pursuit-evasion game with online motion planning by deep reinforcement learning. IEEE Trans Neural Netw Learn Syst2023; 34: 7900–7909.
37.
PengHShenX. Multi-agent reinforcement learning based resource management in MEC- and UAV-assisted vehicular networks. IEEE J Sel Areas Commun2021; 39: 131–141.
38.
WangLWangKPanC, et al.Multi-agent deep reinforcement learning-based trajectory planning for multi-UAV assisted mobile edge computing. IEEE Trans Cognit Commun Networking2021; 7: 73–84.
39.
WuQLinRRenZ. Distributed multirobot path planning based on MRDWA-MADDPG. IEEE Sens J2023; 23: 25420–25432.
40.
WangYZhaoY. Multiple ships cooperative navigation and collision avoidance using multi-agent reinforcement learning with communication. Ocean Eng2025; 320: 120244.
41.
GaoQLiSJiY, et al.Scalable path planning algorithm for multi-unmanned surface vehicles based on multi-agent deep deterministic policy gradient. Ocean Eng2025; 320: 120243.
42.
SchulmanJWolskiFDhariwalP, et al.Proximal policy optimization algorithms. arXiv preprint arXiv:170706347, 2017.
43.
WangZDuJJiangC, et al.Task scheduling for distributed AUV network target hunting and searching: an energy-efficient AoI-aware DMAPPO approach. IEEE Internet Things J2023; 10: 8271–8285.
44.
JiangBDuJJiangC, et al.Underwater searching and multiround data collection via AUV swarms: an energy-efficient AoI-aware MAPPO approach. IEEE Internet Things J2024; 11: 12768–12782.
45.
PuHWangJGaoS, et al.A velocity-domain MAPPO approach for perimeter defensive confrontation by USV groups. Expert Syst Appl2025; 265: 125980.
46.
KangHChangXMišićJ, et al.Cooperative UAV resource allocation and task offloading in hierarchical aerial computing systems: a MAPPO-based approach. IEEE Internet Things J2023; 10: 10497–10509.
47.
GuanYZouSPengH, et al.Cooperative UAV trajectory design for disaster area emergency communications: a multiagent ppo method. IEEE Internet Things J2024; 11: 8848–8859.
48.
QinPFuYZhangJ, et al.DRL-based resource allocation and trajectory planning for noma-enabled multi-UAV collaborative caching 6G network. IEEE Trans Veh Technol2024; 73: 8750–8764.
49.
SunehagPLeverGGruslysA, et al.Value-decomposition networks for cooperative multi-agent learning based on team reward. In: Proceedings of the 17th international conference on autonomous agents and multiagent systems, pp.2085–2087.
50.
SonKKimDKangWJ, et al.QTRAN: learning to factorize with transformation for cooperative multi-agent reinforcement learning. In: International conference on machine learning (ICML), pp.5887–5896.
GuoZWuYWangL, et al.Coordination for connected and automated vehicles at non-signalized intersections: a value decomposition-based multiagent deep reinforcement learning approach. IEEE Trans Veh Technol2023; 72: 3025–3034.
53.
GuoZWuYWangL, et al.Heuristic-based multi-agent deep reinforcement learning approach for coordinating connected and automated vehicles at non-signalized intersection. IEEE Trans Intell Transp Syst2024; 25: 16235–16248.
54.
LinZNiZKuangL, et al.Satellite-terrestrial coordinated multi-satellite beam hopping scheduling based on multi-agent deep reinforcement learning. IEEE Trans Wireless Commun2024; 23: 10091–10103.
55.
XiaoBLiRWangF, et al.Stochastic graph neural network-based value decomposition for MARL in internet of vehicles. IEEE Trans Veh Technol2024; 73: 1582–1596.
GhavamzadehMMahadevanSMakarR. Hierarchical multi-agent reinforcement learning. Auton Agent Multi Agent Syst2006; 13: 197–229.
58.
WuQLiuKChenL, et al.Hierarchical reinforcement learning for swarm confrontation with high uncertainty. IEEE Trans Autom Sci Eng2025; 22: 8630–8644.
59.
ScheidererCMosbachMPosada-MorenoAF, et al.Transfer of hierarchical reinforcement learning structures for robotic manipulation tasks. In: 2020 International conference on computational science and computational intelligence (CSCI), pp.504–509. IEEE.
60.
LiuZCaoYChenJ, et al.A hierarchical reinforcement learning algorithm based on attention mechanism for UAV autonomous navigation. IEEE Trans Intell Transp Syst2022; 24: 13309–13320.
61.
WöhlkeJSchmittFvan HoofH. Hierarchies of planning and reinforcement learning for robot navigation. In: 2021 IEEE international conference on robotics and automation (ICRA), pp.10682–10688. IEEE.
62.
DietterichTG. Hierarchical reinforcement learning with the MAXQ value function decomposition. J Artif Intell Res2000; 13: 227–303.
63.
YanYWangHChenX. Collaborative path planning based on MAXQ hierarchical reinforcement learning for manned/unmanned aerial vehicles. In: 2020 39th Chinese control conference (CCC), pp.4837–4842.
64.
DoroodgarBLiuYNejatG. A learning-based semi-autonomous controller for robotic exploration of unknown disaster scenes while searching for victims. IEEE Trans Cybern2014; 44: 2719–2732.
65.
SuttonRSPrecupDSinghS. Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artif Intell1999; 112: 181–211.
66.
RameshRTomarMRavindranB. Successor options: an option discovery framework for reinforcement learning. arXiv preprint arXiv:190505731, 2019.
67.
ZhangYMouZGaoF, et al.Hierarchical deep reinforcement learning for backscattering data collection with multiple UAVs. IEEE Internet Things J2021; 8: 3786–3800.
68.
ArakiBLiXVodrahalliK, et al.The logical options framework. In: International conference on machine learning (ICML), pp.307–317.
69.
VezhnevetsASOsinderoSSchaulT, et al.Feudal networks for hierarchical reinforcement learning. In: International conference on machine learning (ICML), pp.3540–3549.
70.
LiuXTanY. Feudal latent space exploration for coordinated multi-agent reinforcement learning. IEEE Trans Neural Netw Learn Syst2023; 34: 7775–7783.
71.
JohnsonFCaoBBAshokA, et al.Feudal networks for visual navigation. arXiv preprint arXiv:240212498, 2024.
LevyAKonidarisGPlattR, et al.Learning multi-level hierarchies with hindsight. arXiv preprint arXiv:171200948, 2017.
74.
YangXJiZWuJ, et al.Hierarchical reinforcement learning with universal policies for multistep robotic manipulation. IEEE Trans Neural Netw Learn Syst2022; 33: 4727–4741.
75.
ScheidererCMosbachMPosada-MorenoAF, et al.Transfer of hierarchical reinforcement learning structures for robotic manipulation tasks. In: 2020 International conference on computational science and computational intelligence (CSCI), pp.504–509.
76.
SunCHuangSPompiliD. HMAAC: hierarchical multi-agent actor-critic for aerial search with explicit coordination modeling. In: 2023 IEEE international conference on robotics and automation (ICRA), pp.7728–7734.
77.
NachumOGuSSLeeH, et al.Data-efficient hierarchical reinforcement learning. Adv Neural Inf Process Syst2018; 31: 3303–3313.
78.
Schmoeller RozaFRasheedHRoscherK, et al.Safe robot navigation using constrained hierarchical reinforcement learning. In: 2022 21st IEEE international conference on machine learning and applications (ICMLA), pp.737–742.
79.
HouZFeiJDengY, et al.Data-efficient hierarchical reinforcement learning for robotic assembly control applications. IEEE Trans Ind Electron2021; 68: 11565–11575.
80.
ParrRRussellS. Reinforcement learning with hierarchies of machines. Adv Neural Inf Process Syst1997; 10: 1043–1049.
81.
PanovAISkrynnikA. Automatic formation of the structure of abstract machines in hierarchical reinforcement learning with state clustering. arXiv preprint arXiv:180605292, 2018.
82.
BaconPLHarbJPrecupD. The option-critic architecture. In: Proceedings of the AAAI conference on artificial intelligence, volume 31.
83.
KhetarpalKKlissarovMChevalier-BoisvertM, et al.Options of interest: temporal abstraction with interest functions. In: Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.4444–4451.
84.
ChakravortyJWardNRoyJ, et al.Option-critic in cooperative multi-agent systems. arXiv preprint arXiv:191112825, 2019.
FindikYHasenfusHAzadehR. Collaborative adaptation for recovery from unforeseen malfunctions in discrete and continuous marl domains. arXiv preprint arXiv:240719144, 2024.
87.
ZhaoBHuoMLiZ, et al.Graph-based multi-agent reinforcement learning for large-scale UAVs swarm system control. Aerosp Sci Technol2024; 150: 109166.
88.
QinWBZhangYTakácsD, et al.Nonholonomic dynamics and control of road vehicles: moving toward automation. Nonlinear Dyn2022; 110: 1959–2004.
89.
YangXHuangSSunY, et al.Learning graph-enhanced commander-executor for multi-agent navigation. arXiv preprint arXiv:230204094, 2023.
90.
GuptaLJainRVaszkunG. Survey of important issues in UAV communication networks. IEEE Commun Surv Tutorials2015; 18: 1123–1152.
91.
AlamMMArafatMYMohS, et al.Topology control algorithms in multi-unmanned aerial vehicle networks: an extensive survey. J Network Comput Appl2022; 207: 103495.
92.
GaoAWangQLiangW, et al.Game combined multi-agent reinforcement learning approach for UAV assisted offloading. IEEE Trans Veh Technol2021; 70: 12888–12901.
93.
ZhouY. Efficient online globalized dual heuristic programming with an associated dual network. IEEE Trans Neural Netw Learn Syst2023; 34: 10079–10090.
94.
ZhaoWQueraltaJPWesterlundT. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In: 2020 IEEE symposium series on computational intelligence (SSCI), pp.737–744. IEEE.
95.
JuHJuanRGomezR, et al.Transferring policy of deep reinforcement learning from simulation to reality for robotics. Nat Mach Intell2022; 4: 1077–1087.
96.
KoberJBagnellJAPetersJ. Reinforcement learning in robotics: a survey. Int J Rob Res2013; 32: 1238–1274.
97.
GronauerSDiepoldK. Multi-agent deep reinforcement learning: a survey. Artif Intell Rev2022; 55: 895–943.
98.
BanerjeeCNguyenKFookesC, et al.A survey on physics informed reinforcement learning: review and open problems. arXiv preprint arXiv:230901909, 2023.
99.
CaoYZhaoRWangY, et al.Deep reinforcement learning-based large-scale robot exploration. IEEE Rob Autom Lett2024; 9: 4631–4638.
HuKXuKXiaQ, et al.An overview: attention mechanisms in multi-agent reinforcement learning. Neurocomputing2024; 598: 128015.
102.
ShawSWenzelEWalkerA, et al.Formic: foraging via multiagent RL with implicit communication. IEEE Rob Autom Lett2022; 7: 4877–4884.
103.
WangTBhattDWangX, et al.Cross-embodiment robot manipulation skill transfer using latent space alignment. arXiv preprint arXiv:240601968, 2024.
104.
GuSKubaJGChenY, et al.Safe multi-agent reinforcement learning for multi-robot control. Artif Intell2023; 319: 103905.
105.
GargKZhangSSoO, et al.Learning safe control for multi-robot systems: methods, verification, and open challenges. Annu Rev Control2024; 57: 100948.
106.
MüllerVC. Ethics of artificial intelligence and robotics. In: Zalta EN and Nodelman U (eds) The Stanford encyclopedia of philosophy, Winter 2023 ed. Metaphysics Research Lab, Stanford University, 2023.
107.
WesterlundM. An ethical framework for smart robots. Technol Innovation Manage Rev2020; 10: 35–44.
108.
LeenesRPalmeriniEKoopsBJ, et al.Regulatory challenges of robotics: some guidelines for addressing legal and ethical issues. Law Innovation Technol2017; 9: 1–44.
109.
BennettCRaabC. The governance of privacy: policy instruments in global perspective. Cambridge MA: MIT Press, 2003.
