A real-time energy management strategy for off-road hybrid electric vehicles based on the expected SARSA

Abstract

To improve the power performance and fuel economy and reduce the online computation time of hybrid electric vehicles (HEVs) in off-road driving environments, a real-time energy management strategy based on the expected state-action-reward-state’-action’ (SARSA) algorithm is proposed. This strategy achieves the control effect through coordinated control of the engine generator set (EGS) and the energy distribution between the battery and the EGS. The driving environment is expressed as a transition probability matrix (TPM) of the electric power demand, and the optimal control problem of energy management is solved offline by the expected SARSA algorithm to obtain the control laws. Then, the Kullback-Leibler (KL) divergence rate is used as the evaluation index of the switching control strategy, and these control laws are used online to realize real-time control. Control strategies based on traditional reinforcement learning (RL), stochastic dynamic programming (SDP), and dynamic programming (DP) are considered benchmark strategies to verify the effectiveness of the proposed energy management strategy based on the expected SARSA. The fuel economy is improved by a maximum of 13.2%, and the elapsed time is reduced by more than 99.8% compared with the values achieved with the benchmark algorithms. These results show that the proposed strategy has flexible adaptability to complex and changeable off-road conditions without prior knowledge of the whole driving cycle.

Keywords

Reinforcement learning hybrid vehicle expected SARSA energy management

Get full access to this article

View all access options for this article.

References

. Research on optimization and control of real-time energy management of Mechatronics. Beijing, China, Beijing University of Technology, 2019.

Shabbir

Evangelou

SA.

Exclusive operation strategy for the supervisory control of series hybrid electric vehicles. IEEE Trans Control Syst Technol 2016; 24: 2190–2198.

Xiong

Zhao

, et al. Energy management strategy research on a hybrid power system by hardware-in-loop experiments. Appl Energy 2013; 112: 1311–1317.

Zhang

Xiong

Adaptive energy management of a plug-in hybrid electric vehicle based on driving pattern recognition and dynamic programming. Appl Energy 2015; 155: 68–78.

Rezaei

Burl

Solouk

, et al. Catch energy saving opportunity (CESO), an instantaneous optimal energy management strategy for series hybrid electric vehicles. Appl Energy 2017; 208: 655–665.

Concli

Nezzi

Hybrid transmissions for the optimisation of the efficiency of internal combustion engines. Int J Transp Dev Integr 2020; 4(4): 321–329.

Zeng

Yang

Wang

, et al. Predictive-model-based dynamic coordination control strategy for power-split hybrid electric bus. Mech Syst Signal Process 2015; 60–61: 785–798.

Walker

Zhu

Zhang

Powertrain dynamics and control of a two speed dual clutch transmission for electric vehicles. Mech Syst Signal Process 2017; 85: 1–15.

Zhu

Zhang

Cao

, et al. Robust control of integrated motor-transmission powertrain system over controller area network for automotive applications. Mech Syst Signal Process 2015; 58–59: 15–28.

10.

Xiang

Wang

, et al. Model predictive coordinated control for dual-mode power-split hybrid electric vehicle. Int J Autom Technol 2018; 19(2): 345–358.

11.

Shuai

Gai

, et al. Coordinated motion and powertrain control of a series-parallel hybrid 8 × 8 vehicle with electric wheels. Mech Syst Signal Process 2019; 120: 560–583.

12.

Chen

Xia

You

, et al. A novel energy management method for series plug-in hybrid electric vehicles. Appl Energy 2015; 145: 172–179.

13.

Lee

A supervisory control algorithm for a series hybrid vehicle with multiple energy sources. IEEE Trans Vehicular Technol 2015; 64(11): 4942–4953.

14.

Zhang

Yin

Hardware-in-the-loop simulation of robust mode transition control for a series–parallel hybrid electric vehicle. IEEE Trans Vehicular Technol 2016; 65: 1059–1069.

15.

Chen

Sun

Torque coordination control during mode transition for a series–parallel hybrid electric vehicle. IEEE Trans Vehicular Technol 2012; 61(7): 2936–2949.

16.

Zou

Liu

, et al. Reinforcement learning-based real-time energy management for a hybrid tracked vehicle. Appl Energy 2016; 171: 372–382.

17.

Fechert

Lorenz

Liessner

, et al. Using deep reinforcement learning for hybrid electric vehicle energy management under consideration of dynamic emission models. In: SAE powertrains, fuels and lubricants meeting, Krakow, Poland, 2020.

18.

Serrao

Onori

Rizzoni

A comparative analysis of energy management strategies for hybrid electric vehicles. J Dyn Syst Meas Control 2011; 133(3): 031012.

19.

Zhou

Zhang

, et al. A pseudospectral strategy for optimal power management in series hybrid electric powertrains. IEEE Trans Vehicular Technol 2016; 65(6): 4813–4825.

20.

Chen

, et al. Energy management for a power-split plug-in hybrid electric vehicle based on dynamic programming and neural networks. IEEE Trans Vehicular Technol 2014; 63(4): 1567–1580.

21.

Peng

Xiong

Rule based energy management strategy for a series–parallel plug-in hybrid electric bus optimized by dynamic programming. Appl Energy 2017; 185: 1633–1643.

22.

Chen

Tsai

HC.

Design and analysis of power management strategy for range extended electric vehicle using dynamic programming. Appl Energy 2014; 113: 1764–1774.

23.

Jin

Xiong

Multi-objective optimization study of energy management strategy and economic analysis for a range-extended electric bus. Appl Energy 2017; 194: 798–807.

24.

Mansour

Clodic

Optimized energy management control for the Toyota Hybrid System using dynamic programming on a predicted route with short computation time. Int J Autom Technol 2012; 13(2): 309–324.

25.

Cezar

BLC

Alexandru

. A dynamic programming control strategy for HEV. Appl Mech Mater 2012; 263–266: 541–544.

26.

Inuzuka

Zhang

Shen

Real-Time HEV energy management strategy considering road congestion based on deep reinforcement learning. Energies 2021; 14: 5270.

27.

Xiong

Duan

Cao

, et al. Battery and ultracapacitor in-the-loop approach to validate a real-time power management method for an all-climate electric vehicle. Appl Energy 2018; 217: 153–165.

28.

Peng

, et al. Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus. Appl Energy 2018; 222: 799–811.