This paper presents a unified empirical study of extended Soft Actor-Critic methods for sparse-reward TurtleBot3 navigation in Gazebo under dense 360 LiDAR observations. We introduce SAC-XH, a streamlined SAC extension that augments the sparse task reward with auxiliary shaping signals and integrates a stage-wise curriculum to improve exploration and sample efficiency. Across progressively complex Gazebo environments, SAC-XH improves training stability and success rate compared to SAC, TD3, and DDPG, while maintaining full reproducibility through an open-source ROS 2/Gazebo framework. SAC-XH consistently outperforms the baselines in learning efficiency and success rate, with dense LiDAR observations (360 beams). Additionally, we evaluate a stage-wise Curriculum Learning protocol on top of SAC-XH, using competence-based advancement and controlled replay transfer. Under calibrated thresholds, the curriculum yields stable convergence and high success rates (87–91%), improving generalization across stages compared to non-curriculum training. These results demonstrate that SAC-XH improves convergence and generalization across multiple Gazebo-simulated navigation environments under sparse-reward conditions, providing a strong DRL baseline for autonomous navigation and a reproducible benchmark for future research.
AblettT.ChanB.WangJ. H.KellyJ. (2024). Value-penalized auxiliary control from examples for learning without rewards or demonstrations. https://arxiv.org/abs/2407.03311.
3.
BidakiS. A.MohammadkhahA.RezaeeK.HassaniF.EskandariS.SalahiM.GhassemiM. M. (2025). Online continual learning: A systematic literature review of approaches, challenges, and benchmarks. https://arxiv.org/abs/2501.04897.
4.
de JesusJ. C.KichV. A.KollingA. H.GrandoR. B.de Souza Leite CuadrosM. A.GamarraD. F. T. (2021). Soft actor-critic for navigation of mobile robots. Journal of Intelligent and Robotic Systems: Theory and Applications, 102, 31. https://doi.org/10.1007/s10846-021-01367-5
5.
DongQ.ZengP.HeY.WanG.DongX. (2024). Mitigating catastrophic forgetting in robot continual learning: A guided policy search approach enhanced with memory-aware synapses. IEEE Robotics and Automation Letters, 9(12), 11242–11249. https://doi.org/10.1109/LRA.2024.3487484
6.
FujimotoS.HoofH.MegerD. (2018). Addressing function approximation error in actor-critic methods. In International conference on machine learning (pp. 1587–1596). PMLR.
7.
HaarnojaT.ZhouA.AbbeelP.LevineS. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In J. Dy & A. Krause (Eds.), Proceedings of the 35th international conference on machine learning, Proceedings of Machine Learning Research (Vol. 80, pp. 1861–1870). PMLR. https://proceedings.mlr.press/v80/haarnoja18b.html.
8.
JaziriA.KünzelE.RameshV. (2025). Mitigating the stability-plasticity dilemma in adaptive train scheduling with curriculum-driven continual DQN expansion. https://arxiv.org/abs/2408.09838.
9.
KemkerR.McClureM.AbitinoA.HayesT.KananC. (2018). Measuring catastrophic forgetting in neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). https://doi.org/10.1609/aaai.v32i1.11651
LillicrapT. P.HuntJ. J.PritzelA.HeessN.ErezT.TassaY.SilverD.WierstraD. (2019). Continuous control with deep reinforcement learning. https://arxiv.org/abs/1509.02971.
12.
MirandaV. R. F.NetoA. A.FreitasG. M.MozelliL. A. (2024a). Generalization in deep reinforcement learning for robotic navigation by reward shaping. IEEE Transactions on Industrial Electronics, 71(6), 6013–6020. https://doi.org/10.1109/tie.2023.3290244
13.
MnihV.KavukcuogluK.SilverD.RusuA. A.VenessJ.BellemareM. G.GravesA.RiedmillerM.FidjelandA. K.OstrovskiG.PetersenS.BeattieC.SadikA.AntonoglouI.KingH.KumaranD.WierstraD.LeggS.HassabisD. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529–533. https://doi.org/10.1038/nature14236
14.
MuttaqienM. A.YorozuA.OhyaA. (2024). Mobile robots through task-based human instructions using incremental curriculum learning. https://arxiv.org/abs/2412.19159.
15.
NagarajA.SoodM.PatilB. M. (2022). A concise introduction to reinforcement learning in robotics. https://arxiv.org/abs/2210.07397.
16.
OlayemiK. B.VanM.McLooneS.McIlvannaS.SunY.CloseJ.NguyenN. M. (2023). The impact of LiDAR configuration on goal-based navigation within a deep reinforcement learning framework. Sensors (Basel), 23(24), 9732. https://doi.org/10.3390/s23249732
RemmanS. B.LekkasA. M. (2025). Realistic counterfactual explanations for machine learning-controlled mobile robots using 2d lidar. arXiv preprint arXiv:2505.06906https://arxiv.org/abs/2505.06906.
19.
RengarajanD.VaidyaG.SarveshA.KalathilD.ShakkottaiS. (2022). Reinforcement learning with sparse rewards using guidance from offline demonstration. https://arxiv.org/abs/2202.04628.
20.
RiedmillerM.HafnerR.LampeT.NeunertM.DegraveJ.van de WieleT.MnihV.HeessN.SpringenbergJ. T. (2018). Learning by playing solving sparse reward tasks from scratch. In J. Dy & A. Krause (Eds.), Proceedings of the 35th international conference on machine learning, Proceedings of Machine Learning Research (Vol. 80, pp. 4344–4353). PMLR.
SongS.BihlT.LiuJ. (2025). Coulomb force-guided deep reinforcement learning for effective and explainable robotic motion planning. Frontiers in Robotics and AI, 12, 1697155.
SteinmetzR.RosaF. D.KichV. A.BottegaJ. A.GrandoR. B.GamarraD. F. T. (2025). World models for autonomous navigation of terrestrial robots from lidar observations. Journal of Intelligent & Fuzzy Systems, 18758967251399741. https://doi.org/10.1177/18758967251399741
WangS.TanM.YangZ.WangX.ShenX.HuangH.ZhangW. (2025). Enhancing deep reinforcement learning-based robot navigation generalization through scenario augmentation. https://arxiv.org/abs/2503.01146.
27.
WenS.ShuY.RadA.WenZ.GuoZ.GongS. (2025). A deep residual reinforcement learning algorithm based on soft actor-critic for autonomous navigation. Expert Systems with Applications, 259, 125238.
28.
WisniewskiM.ChatzithanosP.GuoW.TsourdosA. (2025). Benchmarking deep reinforcement learning for navigation in denied sensor environments. Journal of Intelligent & Robotic Systems, 111(3), 103.
29.
XieL.MiaoY.WangS.BlunsomP.WangZ.ChenC.MarkhamA.TrigoniN. (2021). Learning with stochastic guidance for robot navigation. IEEE Transactions on Neural Networks and Learning Systems, 32, 166–176. https://doi.org/10.1109/TNNLS.2020.2977924
30.
YangP.WangX.ZhangR.WangC.OliehoekF. A.KoberJ. (2025). Task-free lifelong robot learning with retrieval-based weighted local adaptation. https://arxiv.org/abs/2410.02995.
31.
ZhangW.WangS.TanM.YangZ.WangX.ShenX. (2025a). DRL-DCLP: A deep reinforcement learning-based dimension-configurable local planner for robot navigation. IEEE Robotics and Automation Letters, 10(4), 3636–3643. https://doi.org/10.1109/LRA.2025.3544927
32.
ZhangZ.FuH.YangJ.LinY. (2025b). Deep reinforcement learning for path planning of autonomous mobile robots in complicated environments. Complex & Intelligent Systems, 11(6), 277.
