Vibration control of three coupled flexible beams using reinforcement learning algorithm based on proximal policy optimization

Abstract

In order to suppress the vibration of a class of multi-coupling flexible beams, a three-coupling flexible beam experimental device is constructed. The vibration characteristics and active vibration control algorithms of three coupled flexible beams are studied. Due to the complex residual vibration of multi-coupled flexible beam structure, which shows the interaction between beams and close frequency characteristics, the dynamic model of coupled flexible beam structure is established by finite element method (FEM) and its theoretical modal analysis is carried out. Furthermore, the parameters of the experimental system are identified, and the close modal frequencies and damping ratios of the actual system are obtained. The characteristics of close modal vibration are verified. A reinforcement learning (RL) vibration controller based on proximal policy optimization (PPO) algorithm is designed. The algorithm mainly includes actor neural network and critic neural network, which are used to approximate the control strategy and evaluate state respectively. Based on theoretical modeling and experimental identification correction model, the agent learns offline, and then transfers the learned policy parameters to the experimental system for vibration control experiments. The experimental results demonstrate that the designed RL controller can effectively suppress the residual vibration of three coupled flexible beams.

Keywords

Coupling flexible beams modal analysis vibration control reinforcement learning proximal policy optimization

Get full access to this article

View all access options for this article.

References

Abdeljaber

Avci

Inman

(2016) Active vibration control of flexible cantilever plates using piezoelectric materials and artificial neural networks. Journal of Sound and Vibration 363: 33–53.

Bandyopadhyay

Manjunath

Umapathy

(2007) Modeling, Control and Implementation of Smart Structures: A FEM-State Space Approach. Berlin Heidelberg: Springer-Verlag.

François-Lavet

Henderson

Islam

, et al. (2018) An introduction to deep reinforcement learning. Foundations and Trends in Machine Learning 11(3–4): 219–354.

Liu

, et al. (2020) Stabilisation of a flexible spacecraft subject to external disturbance and uncertainties. Complexity 2020: 2906546.

Gao

Zhou

, et al. (2021) Reinforcement learning control of a flexible two-link manipulator: An experimental investigation. IEEE Transactions on Systems Man and Cybernetics Systems 51(12): 7326–7336.

Liu

(2020) Distributed vibration control for flexible spacecraft with distributed disturbance and actuator fault. Journal of Sound and Vibration 475: 115274.

Jordan

Mitchell

(2015) Machine learning: Trends, perspectives, and prospects. Science 349(6245): 255–260.

Wang

, et al. (2016) Two-time scale virtual sensor design for vibration observation of a translational flexible-link manipulator based on singular perturbation and differential games. Sensors 16(11): 1804–1817.

Kater

Meurer

(2019) Motion planning and tracking control for coupled flexible beam structures. Control Engineering Practice 84: 389–398.

10.

Lillicrap

Hunt

Pritzel

, et al. (2015) Continuous control with deep reinforcement learning. arXiv e-prints. Available at: http://arxiv.org/abs/1509.02971

11.

Liu

Farimani

(2021) An energy-saving snake locomotion gait policy obtained using deep reinforcement learning. arXiv e-prints. Available at: https://arxiv.org/abs/2103.04511

12.

Liu

Wang

(2018) Disturbance observer based attitude control for flexible spacecraft with input magnitude and rate constraints. Aerospace Science and Technology 72: 486–492.

13.

Long

, et al. (2021) A vibration control method for hybrid-structured flexible manipulator based on sliding mode control and reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems 32(2): 841–852.

14.

Malgaca

Yavuz

Akdağ

, et al. (2016) Residual vibration control of a single-link flexible curved manipulator. Simulation Modelling Practice and Theory 67: 155–170.

15.

Mnih

Kavukcuoglu

Silver

, et al. (2013) Playing Atari with deep reinforcement learning. arXiv e-prints. Available at: https://arxiv.org/abs/1312.5602

16.

Pinto

Andrychowicz

Welinder

, et al. (2017) Asymmetric actor critic for image-based robot learning. arXiv e-prints. Available at: http://arxiv.org/abs/1710.06542

17.

Rong

Rui

, et al. (2011) Discrete time transfer matrix method for dynamic modeling of complex spacecraft with flexible appendages. Journal of Computational and Nonlinear Dynamics 6(1): 011013.

18.

Rosenzweig

Kater

Meurer

(2018) Model predictive control of piezo-actuated structures using reduced order models. Control Engineering Practice 80: 83–93.

19.

Sayahkarajy

Mohamed

Mohd Faudzi

(2016) Review of modelling and control of flexible-link manipulators. Proceedings of the Institution of Mechanical Engineers Part I Journal of Systems and Control Engineering 230(8): 861–873.

20.

Schulman

Levine

Moritz

, et al. (2015a) Trust region policy optimization. arXiv e-prints, 2015. Available at: http://arxiv.org/abs/1502.05477

21.

Schulman

Moritz

Levine

, et al. (2015b) High-dimensional continuous control using generalized advantage estimation. arXiv e-prints. Available at: https://arxiv.org/abs/1506.02438

22.

Schulman

Wolski

Dhariwal

, et al. (2017) Proximal policy optimization algorithms. arXiv e-prints. Available at: http://arxiv.org/abs/1707.06347

23.

Shi

Homberger

Lee

, et al. (2020) Circus ANYmal: A quadruped learning dexterous manipulation with its limbs. arXiv e-prints. Available at: https://arxiv.org/abs/2011.08811

24.

Silver

Lever

Heess

, et al. (2014) Deterministic policy gradient algorithms. In: International conference on machine learning, Vol. 32 (CYCLE 1), Proceedings of machine learning research, Beijing, Peoples Republic of China, 22–24 June.

25.

Sutton

Barto

(2018) Reinforcement Learning: An Introduction, 2nd edn. London: MIT Press.

26.

Van Hasselt

Guez

Silver

(2016) Deep reinforcement learning with double Q-learning. arXiv e-prints. Available at: https://arxiv.org/abs/1509.06461

27.

Wang

(2018) Experiments study on attitude coupling control method for flexible spacecraft. Acta Astronautica 147: 393–402.

28.

Yuan

Liu

(2017) Active vibration suppression for maneuvering spacecraft with high flexible appendages. Acta Astronautica 139: 512–520.

29.

Zhang

Wang

(2019) Tracking control for mobile robot based on deep reinforcement learning. In: 2019 2nd International conference on intelligent autonomous systems (ICoIAS 2019), IEEE, Nanyang Technological University, Singapore, 28 February–2 March, pp.155–160. DOI: 10.1109/ICoIAS.2019.00034.

30.

Zhong

Chen

Guo

(2017) Attitude control for flexible spacecraft with disturbance rejection. IEEE Transactions on Aerospace and Electronic Systems 53(1): 101–110.