Aero-engine acceleration control using deep reinforcement learning with phase-based reward function

Abstract

In this paper, the conventional aero-engine acceleration control task is formulated into a Markov Decision Process (MDP) problem. Then, a novel phase-based reward function is proposed to enhance the performance of deep reinforcement learning (DRL) in solving feedback control tasks. With that reward function, an aero-engine controller based on Trust Region Policy Optimization (TRPO) is developed to improve the aero-engine acceleration performance. Four comparison simulations were conducted to verify the effectiveness of the proposed methods. The simulation results show that the phase-based reward function helps to eliminate the oscillation problem of the aero-engine control system, which is caused by the traditional goal-based reward function when DRL is applied to the aero-engine control. And the TRPO controller outperforms deep Q-learning (DQN) and the proportional-integral-derivative (PID) in the aero-engine acceleration control task. Compared to DQN and PID controller, the acceleration time of aero-engine is decreased by 0.6 and 2.58 s, respectively, and the aero-engine acceleration performance is improved by 16.8 and 46.4% each.

Keywords

Aero-engine acceleration control deep reinforcement learning feedback control system reward function design

Get full access to this article

View all access options for this article.

References

Valasek

. An investigation of fighter aircraft agility. Tech Rep 1993.

Wang

Yue

Liu

. Influence of unmanned combat aerial vehicle agility on short-range aerial combat effectiveness. Aerosp Sci Technol 2020; 96: 105534. DOI: 10.1016/j.ast.2019.105534.

Richter

. Advanced Control of Turbofan Engines. New York, NY: Springer New York, 2012.

Åström

Hägglund

. The future of PID control. Control Eng Pract 2001; 9(11): 1163–1175. DOI: 10.1016/S0967-0661(01)00062-4

Jaw

Mattingly

. Aircraft Engine Controls. Washington, DC: American Institute of Aeronautics and Astronautics, Inc., 2009.

Ashok

Denis Ashok

Ramesh Kumar

. A review on control system architecture of a SI engine management system. Annu Rev Control 2016; 41: 94–118. DOI: 10.1016/j.arcontrol.2016.04.005

Mnih

Kavukcuoglu

Silver

, et al. Human-level control through deep reinforcement learning. Nature 2015; 518(7540): 529–533. DOI: 10.1038/nature14236

Hui

Liu

, et al. Morphing control of a new bionic morphing UAV with deep reinforcement learning. Aerosp Sci Technol 2019; 92: 232–243. DOI: 10.1016/j.ast.2019.05.058

Shalumov

. Cooperative online guide-launch-guide policy in a target-missile-defender engagement using deep reinforcement learning. Aerosp Sci Technol 2020; 104: 105996. DOI: 10.1016/j.ast.2020.105996

10.

Gaudet

Furfaro

Linares

. Reinforcement learning for angle-only intercept guidance of maneuvering targets. Aerosp Sci Technol 2020; 99: 105746. DOI: 10.1016/j.ast.2020.105746

11.

Liu

Tian

, et al. Reinforcement learning based two-level control framework of UAV swarm for cooperative persistent surveillance in an unknown urban area. Aerosp Sci Technol 2020; 98: 105671. DOI: 10.1016/j.ast.2019.105671

12.

Zhang

Wei

. Steady state controller design for aero-engine based on reinforcement learning NNs. In: 29th Chinese Control And Decision Conference (CCDC), Hefei, China, 21–23 May 2022. Chongqing, China: IEEE, 2017, pp. 2168–2173.

13.

Zheng

Jin

, et al. A study of aero-engine control method based on deep reinforcement learning. IEEE Access 2019; 7: 55285–55289. DOI: 10.1109/ACCESS.2018.2883997

14.

Schulman

Levine

Moritz

, et al. Trust Region Policy Optimization. In: Proceedings of the 32nd International Conference on Machine Learning, Lille, France, 07–09 July 2015. PMLR, 2015, pp. 1889–1897.

15.

Harada

Russell

. Policy invariance under reward transformations: Theory and application to reward shaping. In: Proceedings of the Sixteenth International Conference on Machine Learning, ICML ’99, Bled, Slovenia, June 27–30 1999. San Francisco, CA: Morgan Kaufmann Publishers Inc., 1999, pp. 278–287.

16.

Mataric

. Reward functions for accelerated learning. In: Cohen

Hirsh

(eds). Machine Learning Proceedings 1994. Amsterdam, Netherlands: Elsevier, 1994, pp. 181–189.

17.

Matignon

Laurent

Le Fort-Piat

. Reward function and initial values: Better choices for accelerated goal-directed reinforcement learning. In: Kollias

Stafylopatis

Duch

, et al. (eds). Artificial Neural Networks – ICANN 2006, Lecture Notes in Computer Science. Berlin, Heidelberg: Springer; 2006, pp. 840–849.

18.

Sutton

Barto

. Reinforcement Learning: An Introduction. 2nd ed. Cambridge, theUnited Kingdom: The MIT Press, 2018.

19.

Dorf

Bishop

. Modern Control Systems. 12th ed. Hoboken, NJ: PearsonPrentice Hall, 2010.

20.

Burden

Faires

. Numerical Analysis. 9th ed. Boston, MA: Brooks/Cole, Cengage Learning, 2011.

21.

Sutton

Mcallester

Singh

, et al. Policy gradient methods for reinforcement learning with function approximation. Adv Neural Inf Process Syst 1999; 12: 1057–1063.

22.

Kullback

Leibler

. On information and sufficiency. Ann Math Statist 1951; 22(1): 79–86. DOI: 10.1214/aoms/1177729694

23.

Kullback

. Information Theory and Statistics. reprint Edition. Gloucester, MA: Smith, 1978.

24.

Kingma

. Adam: A method for stochastic optimization. CoRR 2015; abs/1412.6980.