This article describes the use of the Deep Deterministic Policy Gradient network, a deep reinforcement learning algorithm, for mobile robot navigation. The neural network structure has as inputs laser range findings, angular and linear velocities of the robot, and position and orientation of the mobile robot with respect to a goal position. The outputs of the network will be the angular and linear velocities used as control signals for the robot. The experiments demonstrated that deep reinforcement learning’s techniques that uses continuous actions, are efficient for decision-making in a mobile robot. Nevertheless, the design of the reward functions constitutes an important issue in the performance of deep reinforcement learning algorithms. In order to show the performance of the Deep Reinforcement Learning algorithm, we have applied successfully the proposed architecture in simulated environments and in experiments with a real robot.
ChenC., SeffA., KornhauserA. and XiaoJ., Deepdriving: Learning affordance for direct perception in autonomous driving, In Proceedings of the IEEE International Conference on Computer Vision (2015), 2722–2730.
2.
ChenY.F., EverettM., LiuM. and HowJ.P., Socially aware motion planning with deep reinforcement learning, In Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ International Conference on, pages 1343–1350. IEEE, (2017).
3.
da SilvaR.M., CuadrosM.A.d.S.L. and GamarraD.F.T., Comparison of a backstepping and a fuzzy controller for tracking a trajectory with a mobile robot. In International Conference on Intelligent SystemsDesign and Applications, pages 212–221. Springer, (2018).
4.
DobrevskiM. and SkocajD., Map-less goal-driven navigation based on reinforcement learning, In 23rd Computer Vision Winter Workshop, (2018).
5.
Dos ReisD.H., WelferD., De Souza Leite CuadrosM.A. and GamarraD.F.T., Mobile robot navigation using an objectrecognition software with rgbd images and the yolo algorithm, Applied Artificial Intelligence33(14) (2019), 1290–1305.
6.
FairchildC. and HarmanT.L., ROS Robotics By Example, Packt Publishing Ltd (2016).
7.
GuS., LillicrapT., SutskeverI. and LevineS., Continuous deep q-learning with model-based acceleration, In International Conference on Machine Learning (2016), 2829–2838.
8.
GuS., HollyE., LillicrapT. and LevineS., Deep reinforcement learning for robotic manipulation with asynchronous offpolicy updates, In 2017 IEEE International Conference on Robotics and Automation (ICRA), 3389–3396. IEEE, (2017).
9.
HausknechtM. and StoneP., Deep recurrent q-learning for partially observable mdps, In 2015 AAAI Fall Symposium Series, (2015).
10.
JesusJ.C., BottegaJ.A., CuadrosM.A. and GamarraD.F., Deep deterministic policy gradient for navigation of mobile robots in simulated environments. In 2019 19th International Conference on Advanced Robotics (ICAR), 362–367. IEEE, (2019).
11.
JosephL., Mastering ROS for robotics programming, Packt Publishing Ltd (2015).
12.
KondaV.R. and TsitsiklisJ.N., Actor-critic algorithms, In Advances in neural information processing systems, (2000), 1008–1014.
13.
LillicrapT.P., HuntJ.J., PritzelA., HeessN., ErezT., TassaY., SilverD. and WierstraD., Continuous control with deep reinforcement learning, arXiv preprint arXiv:1509.02971, (2015).
14.
MahmoodA.R., KorenkevychD., VasanG., MaW. and BergstraJ., Benchmarking reinforcement learning algorithms on real-world robots, arXiv preprint arXiv:1809.07731, (2018).
15.
MirowskiP., PascanuR., ViolaF., SoyerH., BallardA.J., BaninoA., DenilM., GoroshinR., SifreL., KavukcuogluK., et al., Learning to navigate in complex environments, arXiv preprint arXiv:1611.03673, (2016).
16.
MnihV., KavukcuogluK., SilverD., GravesA., AntonoglouI., WierstraD. and RiedmillerM., Playing atari with deep reinforcement learning, arXiv preprint arXiv:1312.5602, (2013).
17.
MnihV., KavukcuogluK., SilverD., RusuA.A., VenessJ., BellemareM.G., GravesA., RiedmillerM., FidjelandA.K., OstrovskiG., et al., Human-level control through deep reinforcement learning, Nature518(7540) (2015), 529–533.
18.
NachumO., NorouziM., XuK. and SchuurmansD., Trustpcl: An off-policy trust region method for continuous control, arXiv preprint arXiv:1707.01891, (2017).
19.
PaszkeA., GrossS., MassaF., LererA., BradburyJ., ChananG., KilleenT., LinZ., GimelsheinN., AntigaL., et al., Pytorch: An imperative style, high-performance deep learning library, In Advances in Neural Information Processing Systems, (2019), 8024–8035.
20.
PfitscherM., WelferD., Do NascimentoE.J., CuadrosM.A.d.S.L. and GamarraD.F.T., Article users activitygesture recognition on kinect sensor using convolutional neural networks and fastdtw for controlling movements ofa mobile robot, Inteligencia Artificial22(63) (2019), 121–134.
QiangL., NanxunD., HuicanL. and HengW., Amodel-free mapless navigation method for mobile robot using reinforcement learning, In 2018 Chinese Control And Decision Conference (CCDC), pages 3410–3415. IEEE, (2018).
23.
RichterF., OroscoR.K. and YipM.C., Open-sourced reinforcement learning environments for surgical robotics, arXiv preprint arXiv:1903.02090, (2019).
SchulmanJ., MoritzP., LevineS., JordanM. and AbbeelP., High-dimensional continuous control using generalized advantage estimation, arXiv preprint arXiv:1506.02438, (2015).
26.
TaiL. and LiuM., Towards cognitive exploration through deep reinforcement learning for mobile robots, arXiv preprint arXiv:1610.01733, (2016).
27.
TaiL., PaoloG. and LiuM., Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation, In Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ International Conference on, pages 31–36. IEEE, (2017).
28.
Tello GamarraD.F., Piccinini LeggA., de Souza Leite CuadrosM.A. and Santos da SilvaE., Sensory integration of a mobile robot using the embedded system odroid-xu4 and ros. In 2019 Latin American Robotics Symposium (LARS), 2019 Brazilian Symposium on Robotics (SBR) and 2019 Workshop on Robotics in Education (WRE), (2019), 198–203.
29.
UhlenbeckG.E. and OrnsteinL.S., On the theory of the brownian motion, Physical Review36(5) (1930), 823.
30.
Van HasseltH., GuezA. and SilverD., Deep reinforcement learning with double q-learning, In Thirtieth AAAI Conference on Artificial Intelligence, (2016).
31.
ZhuY., MottaghiR., KolveE., LimJ.J., GuptaA., Fei-FeiL. and FarhadiA., Target-driven visual navigation in indoor scenes using deep reinforcement learning, In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pages 3357–3364. IEEE, (2017).
