Single-lever control method design based on power management system and deep reinforcement learning for turboprop engines

Abstract

This paper presents a single-lever control method based on Power Management System (PMS) and improved Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm for turboprop engines. In this approach, power level angle command, which is the single-lever command, is decoupled into controlled variable commands by PMS, and the controller based on improved TD3 algorithm can ensure that controlled variables track their commands rapidly and accurately. To achieve the optimal conversion relationship between different commands, an offline optimization process is used to design PMS. By optimization, specific fuel consumption and propeller efficiency are both improved after conversion. To deal with strong interactions between different control loops of a turboprop engine, TD3 algorithm which is a deep reinforcement learning algorithm is adopted. Two improvements which are the design method of observation state and prioritized experience replay are made to enhance the tracking accuracy. Simulation results show that improved TD3 algorithm can learn an optimal control policy to guarantee good control effect with fast response and small overshoot. The maximum settling time is less than 0.25s and the maximum overshoot is less than 0.1%. It also has a good robustness performance when the plant exists model uncertainties. The maximum fluctuations are less than 0.05%.

Keywords

single-lever control power management system deep reinforcement learning turboprop engine feedback control

Get full access to this article

View all access options for this article.

References

Chen

Wang

. Development of turboprop engine control technology in western countries. aeroengine 2016; 42: 9–17, [Chinese].

Europa eagle—a rounded analysis of the Europe large transportation aircraft A400M. Modern Weaponry 2008; (10): 19–30, [Chinese].

Badger

Julien

Leblanc

, et al. The PT6 Engine: 30 Years of Gas Turbine Technology Evolution. J Eng Gas Turbines Power 1993; 116: V001T01A002.

Povazan

Andoga

Fozo

, et al. Introduction to advanced modeling and control of turbo-prop engines. In: IEEE International Conference on Intelligent Engineering Systems, 2012.

Xia

Huang

. Research on power regulation schedule control system for turboprop engine. Int J Turbo Jet Engines 2022; 39(2): 197–204.

Liu

Shi

, et al. Design for aircraft engine multi-objective controllers with switching characteristics. Chin J Aeronaut 2014; 27: 1097–1110.

Le Brun

Godoy

Beauvois

, et al. Control laws design of a turboprop engine. Appl Mech Mater 2015; 704: 362–367.

Chen

Wang

, et al. Inverted decoupling and LMI-based controller design for a turboprop engine with actuator dynamics. Chin J Aeronaut 2020; 33: 1774–1787.

Miao

Xie

Wang

, et al. Terminal sliding mode controller for aeroengine based on RBF neural network. J Aero Power 2010; 25(12): 2281–2286.

10.

Liu

Chen

. Robust adaptive fault-tolerant control for the turbofan aero-engine system. In: 2020 5th International Conference on Advanced Robotics and Mechatronics (ICARM), 2020, pp. 489–494.

11.

Zhou

, et al. An improved multivariable generalized predictive control algorithm for direct performance control of gas turbine engine. Aero Sci Technol 2020; 99: 105576.

12.

Montazeri-Gh

Rasti

Jafari

, et al. Design and implementation of MPC for turbofan engine control system. Aero Sci Technol 2019; 92: 99–113.

13.

Sun

X-M

Wang

Z-M

, et al. A Scheduling scheme of linear model predictive controllers for turbofan engines. IEEE Access 2017; 5: 24533–24541.

14.

Montazeri-Gh

Safari

. Tuning of fuzzy fuel controller for aero-engine thrust regulation and safety considerations using genetic algorithm. Aero Sci Technol 2011; 15: 183–192.

15.

Wang

Sun

, et al. Deterministic policy gradient with integral compensator for robust quadrotor control. IEEE Trans Syst Man Cybern Syst 2020; 50: 3713–3725.

16.

Sutton

Barto

Williams

. Reinforcement learning is direct adaptive optimal control. IEEE Control Syst Mag 1992; 12: 19–22.

17.

Silver

Huang

Maddison

, et al. Mastering the game of Go with deep neural networks and tree search. Nature 2016; 529: 484–489.

18.

Hasselt

Guez

Silver

. Deep Reinforcement Learning with Double Q-Learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 30, 2016, https://ojs.aaai.org/index.php/AAAI/article/view/10295

19.

Lillicrap

Hunt

Pritzel

, et al. Continuous control with deep reinforcement learning, 2019.

20.

Fujimoto

Hoof

Meger

. Addressing function approximation error in actor-critic methods. In: Proceedings of the 35th International Conference on Machine Learning. PMLR, 2018; 1587–1596.

21.

Hwangbo

Siegwart

, et al. Control of a quadrotor with reinforcement learning. IEEE Rob Autom Lett 2017; 2: 2096–2103.

22.

Zheng

Jin

, et al. A study of aero-engine control method based on deep reinforcement learning. IEEE Access 2019; 7: 55285–55289.

23.

Q-K

Zhao

Y-P

. Aero-engine acceleration control using deep reinforcement learning with phase-based reward function. Proc Inst Mech Eng Part G J Aerosp Eng 2022; 236(9): 1878–1894.

24.

Chen

Wang

. A modeling method of propeller based on the propeller component characteristic. J Aero Power 2017; 32: 2526–2535, [Chinese].

25.

Liu

Y-J

Cheng

S-M

Hsueh

Y-L

. eNB selection for machine type communications using reinforcement learning based markov decision process. IEEE Trans Veh Technol 2017; 66: 11330–11338.

26.

Duan

Shi

Diao

, et al. Deep-reinforcement-learning-based autonomous voltage control for power grid operations. IEEE Trans Power Syst 2020; 35: 814–817.

27.

Wang

F-Y

Zhang

Liu

. Adaptive dynamic programming: an introduction. IEEE Comput Intell Mag 2009; 4: 39–47.

28.

Radac

M-B

Precup

R-E

. Data-driven model-free tracking reinforcement learning control with VRFT-based adaptive actor-critic. Appl Sci 2019; 9: 1807.

29.

Zhou

Xue

, et al. A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning. Energy 2021; 224: 120118.

30.

Adam

Busoniu

Babuska

. Experience replay for real-time reinforcement learning control. IEEE Trans Syst Man Cybern C Appl Rev 2012; 42: 201–212.