Passive dynamic walking (PDW) has attracted much research attention due to its humanoid and energy efficient gaits. However, walking control of the passivity-based biped robot inspired by PDW still remains a challenge, for PDW is sensitive to disturbances. An walking controller is essential for practical passivity-based biped robots in real environments. This paper presents a deep reinforcement learning (DRL) controller based on deep Q network for planar passivity-based biped robot, to learn policies directly from inputs for bipedal walking task. First, the intelligent controller using deep Q network is trained, with PDW as reference trajectory. The learning experience from PDW could be helpful to implement a natural looking and energy-efficient gait. Then the trained deep Q network is utilized as the walking controller. Simulation results show that the DRL controller based on deep Q network makes the planar biped robot walk against original value disturbance, on different slope, level ground and varying slopes. The controller this paper presented could be used to improve the versatility of the passivity-based biped robot.
SakagamiY., WatanabeR., AoyamaC., MatsunagaS. and FujimuraK., The intelligent ASIMO: System overview and integration, In Intelligent Robots and Systems, 2002, IEEE/RSJ International Conference on, 2002, pp. 2478–2483.
2.
KanekoK., KanehiroF., MorisawaM. and AkachiK., Humanoid robot HRP-4 - Humanoid robotics platform with lightweight and slim body, In Intelligent Robots and Systems (IROS), 2011 IEEE/RSJ International Conference on, 2011, pp. 4400–4407.
3.
VukobratovićM. and StepanenkoJ., On the stability of anthropomorphic systems, Mathematical Biosciences15 (1972), 1–37.
4.
CollinsS., RuinaA., TedrakeR. and WisseM., Efficient bipedal robots based on passive-dynamic walkers, Science307 (2005), 1082–1085.
5.
WuY., YaoD. and XiaoX., The effects of ground compliance on flexible planar passive biped dynamic walking, Journal of Mechanical Science and Technology32 (2018), 1793–1804.
6.
McGeerT., Passive dynamic walking, The International Journal of Robotics Research9 (1990), 62–82.
7.
SpongM.W., HolmJ.K. and LeeD., Passivity-based control of bipedal locomotion, IEEE Robotics & Automation Magazine14 (2007), 30–40.
8.
WangS., ChaovalitwongseW. and BabuskaR., Machine learning algorithms in bipedal robot control, IEEE Transactions on Systems Man and Cybernetics42 (2012), 728–743.
9.
SuttonR.S., BartoA.G., Reinforcement learning: An introduction, Cambridge: MIT Press, 1998.
10.
WatkinsC.J. and DayanP., Q-learning, Machine Learning8 (1992), 279–292.
11.
RummeryG.A., NiranjanM., On-line Q-learning using connectionist systems, University of Cambridge, Department of Engineering1994.
12.
PengJ. and WilliamsR.J., Incremental multi-step Q-learning, Machine learning22 (1996), 283–290.
13.
WangS., BraaksmaJ., BabuskaR. and HobbelenD., Reinforcement learning control for biped robot walking on uneven surfaces, In Neural Networks, 2006, IJCNN’06, International Joint Conference on, 2006, pp. 4173–4178.
14.
DavariM., AlipourK., HadiA. and TarvirdizadehB., Learning a model-free robotic continuous state-action task through contractive Q-network, In Artificial Intelligence and Robotics (IRANOPEN) (2017), 115–120.
15.
WuC., SongH., YanC. and WangY., A fuzzy-based function approximation technique for reinforcement learning, Journal of Intelligent & Fuzzy Systems32 (2017), 3909–3920.
16.
LuvianoD. and YuW., Continuous-time path planning for multi-agents with fuzzy reinforcement learning, Journal of Intelligent & Fuzzy Systems33 (2017), 491–501.
17.
TedrakeR., ZhangT.W. and SeungH.S., Stochastic policy gradient reinforcement learning on a simple 3D biped, Intelligent Robots and Systems, 2004 (IROS 2004) Proceedings 2004 IEEE/RSJ International Conference on, 2004, pp. 2849–2854.
18.
FeirsteinD.S., KoryakovskiyI., KoberJ. and ValleryH., Reinforcement learning of potential fields to achieve limit-cycle walking, IFAC-Papers OnLine49 (2016), 113–118.
19.
LiC., LoweR. and ZiemkeT., A novel approach to locomotion learning: Actor-Critic architecture using central pattern generators and dynamic motor primitives, Frontiers in Neurorobotics8 (2014), 1–17.
20.
BengioY., Learning deep architectures for AI, Foundations & Trends in Machine Learning2 (2009), 1–127.
21.
ArulkumaranK., DeisenrothM.P., BrundageM. and BharathA.A., Deep reinforcement learning: A brief survey, IEEE Signal Processing Magazine34 (2017), 26–38.
22.
MnihV., KavukcuogluK., SilverD., GravesA., AntonoglouI., WierstraD. and RiedmillerM., Playing Atari with Deep Reinforcement Learning, 2013, arXiv:1312.5602.
23.
ZhangH., SaninC., SzczerbickiE. and ZhuM., Towards neural knowledge DNA, Journal of Intelligent & Fuzzy Systems32 (2017), 1575–1584.
24.
SilverD., HuangA., MaddisonC.J., GuezA., SifreL. and DielemanS., Mastering the game of Go with deep neural networks and tree search, Nature529 (2016), 484–489.
25.
SilverD., SchrittwieserJ., SimonyanK., AntonoglouI., HuangA., GuezA. and ChenY., Mastering the game of Go without human knowledge, Nature550 (2017), 354–359.
26.
LiY., Deep reinforcement learning: An overview, 2017, arXiv: 1701.07274.
27.
MnihV., KavukcuogluK., SilverD., RusuA.A., VenessJ., BellemareM.G. and PetersenS., Human-level control through deep reinforcement learning, Nature518 (2015), 529–533.
28.
Van HasseltH., GuezA. and SilverD., Deep Reinforcement Learning with Double Q-Learning, In AAAI (2016), 2094–2100.
29.
GoswamiA., ThuilotB. and EspiauB., A study of the passive gait of a compass-like biped robot: Symmetry and chaos, The International Journal of Robotics Research17 (1998), 1282–1301.
30.
OrY. and MoraviaM., Analysis of foot slippage effects on an actuated spring-mass model of dynamic legged locomotion, International Journal of Advanced Robotic Systems13 (2016), 69.
31.
SafartoobiM., DardelM., GhasemiM.H. and DanialiH.M., Determination of the initial conditions by solving boundary value problem method for period-one walking of a passive biped walking robots, Robotica35 (2017), 166–188.
32.
KwanM. and HubbardM., Optimal foot shape for a passive dynamic biped, Journal of Theoretical Biology248 (2007), 331–339.
33.
NingL., JunfengL. and TianshuW., The effects of parameter variation on the gaits of passive walking models: Simulations and experiments, Robotica27 (2009), 511–528.
34.
WesterveltE.R., GrizzleJ.W., ChevallereauC., ChoiJ.H., MorrisB., Feedback control of dynamic bipedal robot locomotion, CRC Press, 2007.
35.
BatheK.J., Finite element procedures in engineering analysis, Prentice-Hall, 1982.
36.
EberhardP., HuB., Advanced contact dynamics, Southeast University Press, 2003.
37.
LinL.J., Carnegie-Mellon Univ Pittsburgh PA School of Computer Science, Reinforcement learning for robots using neural networks, 1993.
38.
AbadiM., AgarwalA., BarhamP., BrevdoE., ChenZ., CitroC. and GhemawatS., Tensorflow: Large-scale machine learning on heterogeneous distributed systems, 2016, arXiv: 1603.04467.
39.
LiuL., JiangH., LiJ., TianY. and YangM., Adaptive excitation control for the underactuated biped robot, Energy Procedia17 (2012), 1667–1674.
