The safe and efficient navigation of mobile robots in the presence of unknown dynamic obstacles remains a complex and unresolved challenge. This paper presents collision-free path planning for a mobile robot that safely deals with multi-directional obstacles, that is, randomly moving dynamic obstacles, using a Deep Reinforcement Learning (DRL) algorithm named Deep Q-Network (DQN) with inflated robot reward functions. The robot moves in a time-efficient and collision-free route while maintaining a safe distance with both static and unpredictable dynamic obstacles. The modified DQN algorithm takes RGB images of the environment as input for training a Convolution Neural Network (CNN) and provides a safe and short path for navigation. The robot used for training is an omni-wheeled mobile robot exploring outdoors, that is, concourse environment, and indoors, that is, home environment. The Closed-Loop Inverse Kinematics (CLIK) algorithm is employed to control a mobile robot to follow the desired path. The simulation results indicate that the proposed algorithm with inflated robot reward functions demonstrates remarkable performance as compared to recently used Reinforcement Learning (RL) algorithms when dealing with both stationary and randomly moving obstacles in the given environment.
BonnyTKashkashM.Highly optimized Q-learning-based bees approach for mobile robot path planning in static and dynamic environments. J Field Robot2022; 39: 317–334.
2.
HarwinLSupriyaP. Comparison of SARSA algorithm and temporal difference learning algorithm for robotic path planning for static obstacles. In: 2019 third international conference on inventive systems and control (ICISC), 2019, pp.472–476. New York: IEEE.
3.
MnihVKavukcuogluKSilverD, et al. Human-level control through deep reinforcement learning. Nature2015; 518: 529–533.
4.
GaoXYanLLiZ, et al. Improved deep deterministic policy gradient for dynamic obstacle avoidance of mobile robot. IEEE Trans Syst Man Cybern Syst2023; 53(6): 3675–3682.
5.
SachanSPathakPM. Path planning and control for omni-directional mobile robot using Q-learning and CLIK algorithm for home environment. In: Cal-Tek srl, 2022. Epub ahead of print 2022. DOI: 10.46354/i3m.2022.mas.027.
6.
ZhangLWuJLiuC, et al. A novel control strategy of automatic parallel parking system based on Q-learning. Proc IMechE, Part D: J Automobile Engineering2024; 238: 661–673.
7.
DuguleanaMMoganG.Neural networks based reinforcement learning for mobile robots obstacle avoidance. Expert Syst Appl2016; 62: 104–115.
8.
LowESOngPCheahKC.Solving the optimal path planning of a mobile robot using improved Q-learning. Rob Auton Syst2019; 115: 143–161.
9.
ZhouQLianYWuJ, et al. An optimized Q-Learning algorithm for mobile robot local path planning. Knowl Based Syst2024; 286: 111400.
10.
WangPLiXSongC, et al. Research on dynamic path planning of wheeled robot based on deep reinforcement learning on the slope ground. J Robot2020; 1: 7167243.
11.
WenSZhaoYYuanX, et al. Path planning for active SLAM based on deep reinforcement learning under unknown environments. Intell Serv Robot2020; 13: 263–272.
12.
YuJSuYLiaoY.The path planning of mobile robot by neural networks and hierarchical reinforcement learning. Front Neurorobot2020; 14: 63.
13.
LilZXinlJLilN. Autonomous exploration and mapping for mobile robots via cumulative curriculum reinforcement learning. In: 2023 IEEE/RSJ international conference on intelligent robots and systems (IROS), 2023, pp.7495–7500. New York: IEEE.
14.
ManhTNManhCNTienDP, et al. Autonomous navigation for omnidirectional robot based on deep reinforcement learning. Int J Mech Eng Robot Res2020; 9: 1134–1139.
15.
KumaarAANKochuvilaS. Mobile service robot path planning using deep reinforcement learning. IEEE Access2023; 8: 1–14.
16.
Van HasseltHGuezASilverD. Deep reinforcement learning with double q-learning. In: Proceedings of the AAAI conference on artificial intelligence, 2016, vol. 30, no. 1. New York: ASME.
17.
TaiLLiSLiuM. A deep-network solution towards model-less obstacle avoidance. In: 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS), 2016, pp.2759–2764. New York: IEEE.
18.
ZhouSLiuXXuY, et al. A deep Q-network (DQN) based path planning method for mobile robots. In: 2018 IEEE international conference on information and automation (ICIA), 2018, pp.366–371. New York: IEEE.
19.
JiangLHuangHDingZ.Path planning for intelligent robots based on deep Q-learning with experience replay and heuristic knowledge. IEEE/CAA J Automat Sinica2019; 7: 1179–1189.
20.
LiuSChangPLiangW, et al. Decentralized structural-RNN for robot crowd navigation with deep reinforcement learning. In: 2021 IEEE international conference on robotics and automation (ICRA), 2021, pp.3517–3524. New York: IEEE.
21.
ZhouZZhuPZengZ, et al. Robot navigation in a crowd by integrating deep reinforcement learning and online planning. Appl Intell2022; 52: 15600–15616.
22.
ZhangTLiuZPuZ, et al. Robot subgoal-guided navigation in dynamic crowded environments with hierarchical deep reinforcement learning. Int J Control Autom Syst2023; 21: 2350–2362.
23.
MonteroEEMutahiraHPicoN, et al. Dynamic warning zone and a short-distance goal for autonomous robot navigation using deep reinforcement learning. Complex Intell Syst2024; 10: 1149–1166.
24.
ZhouZZengZLangL, et al. Navigating robots in dynamic environment with deep reinforcement learning. IEEE Trans Intell Transport Syst2022; 23: 25201–25211.
25.
Martinez-BaselgaDRiazueloLMontanoL. Improving robot navigation in crowded environments using intrinsic rewards. arXiv preprint arXiv:230206554.
26.
XuJZhangWCaiJ, et al. SafeCrowdNav: safety evaluation of robot crowd navigation in complex scenes. Front Neurorobot2023; 17: 1276519.
27.
ChoiJLeeGLeeC.Reinforcement learning-based dynamic obstacle avoidance and integration of path planning. Intell Serv Robot2021; 14: 663–677.
28.
PhueakthongPVaragulJPinrathN. Deep reinforcement learning based mobile robot navigation in unknown environment with continuous action space. In: 2022 5th international conference on intelligent autonomous systems (ICoIAS), 2022, pp.154–158. New York: IEEE.
29.
AzarATSardarMZAhmedS, et al. Autonomous robot navigation and exploration using deep reinforcement learning with Gazebo and ROS. In: International conference on advanced intelligent systems and informatics, 2023, pp.287–299. New York: IEEE.
30.
ChangLShanLZhangW, et al. Hierarchical multi-robot navigation and formation in unknown environments via deep reinforcement learning and distributed optimization. Robot Comput Integr Manuf2023; 83: 102570.
31.
GongHWangPNiC, et al. Efficient path planning for mobile robot based on deep deterministic policy gradient. Sensors2022; 22: 3579.
32.
YangLBiJYuanH.Dynamic path planning for mobile robots with deep reinforcement learning. IFAC-PapersOnLine2022; 55: 19–24.
33.
ParkK-WKimMKimJ-S, et al. Path planning for multi-arm manipulators using soft actor-critic algorithm with position prediction of moving obstacles via LSTM. Appl Sci2022; 12: 9837.
34.
NakraniNMJoshiMM.A human-like decision intelligence for obstacle avoidance in autonomous vehicle parking. Appl Intell2022; 52(4): 3728–3747.
35.
ZhangRXuXZhangX, et al. Kernel-based multiagent reinforcement learning for near-optimal formation control of mobile robots. Appl Intell2023; 53: 12736–12748.
36.
DangTVBuiNT.Multi-scale fully convolutional network-based semantic segmentation for mobile robot navigation. Electronics (Basel)2023; 12: 533.
37.
SunRHZhaoXWuCD, et al. Research on mobile robot navigation method based on semantic information. Sensors2024; 24: 4341.
38.
ChenWZhouSPanZ, et al. Mapless collaborative navigation for a multi-robot system based on the deep reinforcement learning. Appl Sci2019; 9: 4198.
39.
SyedUAKunwarFIqbalM.Guided autowave pulse coupled neural network (GAPCNN) based real time path planning and an obstacle avoidance scheme for mobile robots. Robot Auton Syst2014; 62: 474–486.
40.
ChoYManzoorSChoiY.Adaptation to environmental change using reinforcement learning for robotic salamander. Intell Serv Robot2019; 12: 209–218.
41.
VemulaAMuellingKOhJ. Social attention: modeling attention in human crowds. In: 2018 IEEE international conference on robotics and automation (ICRA), 2018, pp.4601–4607. New York: IEEE.
42.
LowESOngPLowCY, et al. Modified Q-learning with distance metric and virtual target on path planning of mobile robot. Expert Syst Appl2022; 199: 117191.
43.
XuSHaoJChenX, et al. Navigating autonomous vehicles in uncertain environments with distributional reinforcement learning. Proc IMechE, Part D: J Automobile Engineering. Epub ahead of print 2023. DOI: 10.1177/0954407023118684.
44.
RamRVPathakPMJuncoSJ.Trajectory control of a mobile manipulator in the presence of base disturbance. Simulation2019; 95: 529–543.
45.
LeiXZhangZDongP.Dynamic path planning of unknown environment based on deep reinforcement learning. J Robot2018; 1: 5781591.
46.
Do QuangHManhTNManhCN, et al. An approach to design navigation system for omnidirectional mobile robot based on ROS. Int J Mech Eng Robot Res2020; 9: 1502–1508.
47.
ChikurtevD. Mobile robot simulation and navigation in ROS and Gazebo. In: 2020 International conference automatics and informatics (ICAI), 2020, pp.1–6. New York: IEEE.
48.
JiangHWanK-WWangH, et al. A dueling twin delayed DDPG architecture for mobile robot navigation. In: 2022 17th international conference on control, automation, robotics and vision (ICARCV), 2022, pp.193–197. New York: IEEE.
49.
SuttonRSBartoAG.Reinforcement learning: an introduction. London: MIT Press, 2018.
50.
LapanM.Deep Reinforcement learning hands-on: apply modern RL methods, with deep Q-networks, value iteration, policy gradients, TRPO, AlphaGo Zero and more. Birmingham: Packt Publishing, 2018.
51.
RayankulaVPathakPM.Fault tolerant control and reconfiguration of mobile manipulator. J Intell Robot Syst2021; 101: 1–18.
52.
ZhangYChenP.Path planning of a mobile robot for a dynamic indoor environment based on an SAC-LSTM algorithm. Sensors2023; 23: 9802.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
