Single-track two-wheeled robots have become an important research topic in recent years, owing to their simple structure, energy savings and ability to run on narrow roads. However, the ramp jump remains a challenging task. In this study, we propose to realize a single-track two-wheeled robot ramp jump. We present a control method that employs continuous action reinforcement learning techniques for single-track two-wheeled robot control. We design a novel reward function for reinforcement learning, optimize the dimensions of the action space, and enable training under the deep deterministic policy gradient algorithm. Finally, we validate the control method through simulation experiments and successfully realize the single-track two-wheeled robot ramp jump task. Simulation results validate that the control method is effective and has several advantages over high-dimension action space control, reinforcement learning control of sparse reward function and discrete action reinforcement learning control.
AstromKJKleinRELennartssonA (2005) Bicycle dynamics and control: Adapted bicycles for education and research. IEEE Control Systems Magazine25(4): 26–47.
2.
BeznosAVFormal’SkyAMGurfinkelEV, et al. (1998) Control of autonomous motion of two-wheel bicycle with gyroscopic stabilisation. In: Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No. 98CH36146), Vol. 3, Leuven, Belgium, 16–20 May 1998, Piscataway, NJ, pp. 2670–2675. IEEE.
3.
BhatnagarSSuttonRS (2007) Incremental natural actor-critic algorithms. In: Proceedings of the 21st Annual Conference on Neural Information Processing Systems, Vancouver, Canada, 3–6 Dec. 2007, San Diego, CA, 105, 112.
4.
ChenLLiuJWangH, et al. (2021) Robust control of reaction wheel bicycle robot via adaptive integral terminal sliding mode. Nonlinear Dynamics: 104(3), 2291–2302.
5.
ChoiSLeTPNguyenQD, et al. (2019) Toward self-driving bicycles using state-of-the-art deep reinforcement learning algorithms. Symmetry11(2): 290. DOI: https://doi.org/10.3390/sym11020290
GetzNH (1994) Control of balance for a nonlinear nonholonomic non-minimum phase model of a bicycle. In: Proceedings of 1994 American Control Conference-ACC’94, Vol. 1, Baltimore, MD, USA, 29 June–1 July 1994, Piscataway NJ, pp. 148–151. IEEE.
8.
GetzNH (1995) Internal equilibrium control of a bicycle. In: Proceedings of 1995 34th IEEE Conference on Decision and Control, Vol. 4, New Orleans, LA, USA, 13–15 Dec. 1995, Piscataway, NJ, pp. 4285–4287. IEEE.
9.
HausknechtMStoneP (2015) Deep recurrent q-learning for partially observable mdps. arXiv preprint arXiv:1507.06527.
10.
HeJZhaoM (2015) Control system design of self-balanced bicycles by control moment gyroscope. In: Proceedings of the 2015 Chinese Intelligent Automation Conference, Fuzhou, China, Jan-15, pp. 205–214. Berlin, Heidelberg: Springer.
11.
HuangYLiaoQGuoL, et al. (2015) Simple realization of balanced motions under different speeds for a mechanical regulator-free bicycle robot. Robotica33(9): 1958–1972.
12.
HwangCLWuHMShihCL (2009) Fuzzy sliding-mode underactuated control for autonomous dynamic balance of an electrical bicycle. IEEE Transactions on Control Systems Technology17(3): 658–670.
13.
KeoLYamakitaM (2011) Control of an autonomous electric bicycle with both steering and balancer controls. Advanced Robotics25(1–2): 1–22.
14.
KimYKimHLeeJ (2015) Stable control of the bicycle robot on a curved path by using a reaction wheel. Journal of Mechanical Science and Technology29(5): 2219–2226.
15.
LeeSHamW (2002) Self stabilizing strategy in tracking control of unmanned electric bicycle with mass balance. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, Vol. 3, Lausanne, Switzerland, 30 Sept.–4 Oct. 2002, Piscataway, NJ, pp. 2200–2205. IEEE.
16.
LillicrapTPHuntJJPritzelA, et al. (2015) Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971.
17.
LinLJ (1992) Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning8(3–4): 293–321.
18.
LiptonZCLiXGaoJ, et al. (2016) Efficient dialogue policy learning with bbq-networks. arXiv preprint ArXiv:1608.05081.
19.
MahajanATulabandhulaT (2017) Symmetry learning for function approximation in reinforcement learning. arXiv preprint arXiv:1706.02999.
20.
MnihVKavukcuogluKSilverD, et al. (2013) Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602.
21.
MnihVKavukcuogluKSilverD, et al. (2015) Human-level control through deep reinforcement learning. Nature518(7540): 529–533.
22.
MnihVBadiaAPMirzaM, et al. (2016) Asynchronous methods for deep reinforcement learning. In: ICML’16 Proceedings of the 33rd International Conference on International Conference on Machine Learning, Vol. 48, pp. 1928–1937.
SeekhaoPTungpimolrutKParnichkunM (2020) Development and control of a bicycle robot based on steering and pendulum balancing. Mechatronics69: 102386. DOI: 10.1016/J.MECHATRONICS.2020.102386
25.
SchaulTQuanJAntonoglouI, et al. (2015) Prioritized experience replay. arXiv preprint arXiv:1511.05952.
26.
SilverDLeverGHeessN, et al. (2014). Deterministic Policy Gradient Algorithms. In: Proceedings of The 31st International Conference on Machine Learning. pp. 387–395.
27.
SunYZhaoMWangB, et al. (2020) Polynomial controller for bicycle robot based on nonlinear descriptor system. In: IECON 2020 The 46th Annual Conference of the IEEE Industrial Electronics Society, Singapore, 18–21 Oct. 2020, Piscataway, NJ, pp. 2792–2797. IEEE.
28.
SuttonRS (1988) Learning to predict by the methods of temporal differences. Machine Learning3(1): 9–44.
29.
SuttonRSMcAllesterDSinghS, et al. (1999) Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems12: 1057–1063.
30.
TanakaYMurakamiT (2004) Self sustaining bicycle robot with steering controller. In: The 8th IEEE International Workshop on Advanced Motion Control, 2004, AMC’04, Kawasaki, Japan, 28 March 2004, Piscataway, NJ, pp. 193–197. IEEE.
31.
TanakaYMurakamiT (2009) A study on straight-line tracking and posture control in electric bicycle. IEEE Transactions on Industrial Electronics56(1): 159–168.
32.
Van HasseltHGuezASilverD (2015) Deep reinforcement learning with double q-learning. arXiv preprint arXiv:1509.06461.
33.
WangPGongYYiJ, et al. (2019a) An integrated stationary/moving balance control of an autonomous bikebot. In: 2019 American Control Conference (ACC), Philadelphia, PA, USA, 10–12 July 2019, Piscataway, NJ, pp. 3273–3278. IEEE.
34.
WangPYiJLiuT (2019b) Stability and control of a rider–bicycle system: Analysis and experiments. IEEE Transactions on Automation Science and Engineering17(1): 348–360.
35.
WangZWangYZhangB, et al. (2020) Development of a two-wheel steering unmanned bicycle: simulation and experimental study. In: 2020 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), Boston, MA, USA, 6–9 July 2020Piscataway, NJ, pp. 119–124. IEEE.
YamakitaMUtanoA (2005) Automatic control of bicycles with a balancer. In: Proceedings, 2005 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, Monterey, CA, USA, 24–28 July 2005, Piscataway, NJ, pp. 1245–1250.
38.
YamakitaMUtanoASekiguchiK (2006) Experimental study of automatic control of bicycle with balancer. In: 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, Beijing, China, 9–15 Oct. 2006, Piscataway, NJ, pp. 5606–5611. IEEE.
39.
YetkinHKaloucheSVernierM, et al. (2014) Gyroscopic stabilization of an unmanned bicycle. In: 2014 American Control Conference, Portland, OR, USA, 4–6 June 2014, Piscataway, NJ, pp. 4549–4554. IEEE.
40.
YuYZhaoM (2018) Steering control for autonomously balancing bicycle at low speed. In: 2018 IEEE International Conference on Robotics and Biomimetics (ROBIO), Kuala Lumpur, Malaysia, 12–15 Dec. 2018, Piscataway, NJ, pp. 33–38. IEEE.
41.
ZhangYLiJYiJ, et al. (2011) Balance control and analysis of stationary riderless motorcycles. In: 2011 IEEE International Conference on Robotics and Automation, Shanghai, China, 9–13 May 2011, Piscataway, NJ, pp. 3018–3023. IEEE.
