Energy management strategy for hybrid electric propulsion system based on parametric deep Q-network in hybrid action space

Abstract

This study proposes an energy management strategy (EMS) for hybrid power systems based on model-free deep reinforcement learning using parametric Deep Q-Network (P-DQN/MP-DQN). The traditional deep reinforcement learning (DRL) paradigm-based EMS is limited by its self-learning capabilities, making it unable to simultaneously achieve hybrid action control of shifting rules and overall vehicle EMS. The EMS using DDPG (Deep Deterministic Policy Gradient) with a separately designed single-parameter shifting rule achieves about 94% of the fuel economy of single-parameter DP (Dynamic Programming). There remains significant room for improvement in DRL-based EMS that includes transmission shifting strategies. To address this, we employed a parametric action space to enhance hybrid action control in the EMS and used a Multi-Pass Q-network (MP-QN) to decouple the coherence of Q-values for discrete actions. The multi-channel architecture of MP-QN enables each channel to independently learn and optimize the Q value of a specific action, enhancing the ability to handle complex decision-making processes. The results show that the EMS based on P-DQN and MP-DQN formulated an excellent intelligent shifting rule similar to multi-parameter DP (DP-m) and significantly improved global optimality. Compared to the EMS based on single-parameter shifting rule DP (DP-s), fuel economy is improved. Additionally, the MP-DQN using the Multi-Pass Q-network significantly improved the stability and fuel economy of EMS training convergence, demonstrating its excellent potential for hybrid action control.

Keywords

Energy management strategy hybrid power systems model-free deep reinforcement learning parametric DQN hybrid action control

Get full access to this article

View all access options for this article.

References

Zhu

Song

, et al. Hierarchical reinforcement learning based energy management strategy for hybrid electric vehicle. Energy 2022; 238: 121703.

Hannan

Azidin

Mohamed

Hybrid electric vehicles and their challenges: a review. Renew Sustain Energy Rev 2014; 29: 135–150.

Luján

Guardiola

Pla

, et al. Analytical optimal solution to the energy management problem in series hybrid electric vehicles. IEEE Trans Veh Technol 2018; 67(8): 6803–6813.

Williamson

SS.

Energy management strategies for electric and plug-in hybrid electric vehicles. New York, NY: Springer, 2013.

Zou

Kong

Liu

, et al. A real-time Markov chain driver model for tracked vehicles and its validation: its adaptability via stochastic dynamic programming. IEEE Trans Veh Technol 2016; 66(5): 3571–3582.

Sabri

MFM

Danapalasingam

Rahmat

MF.

A review on hybrid electric vehicles architecture and energy management strategies. Renew Sustain Energy Rev 2016; 53: 1433–1442.

Tie

Tan

CW.

A review of energy sources and energy management system in electric vehicles. Renew Sustain Energy Rev 2013; 20: 82–102.

Zhang

Xiong

Adaptive energy management of a plug-in hybrid electric vehicle based on driving pattern recognition and dynamic programming. Appl Energy 2015; 155: 68–78.

Chen

Xiong

, et al. Energy management of a power-split plug-in hybrid electric vehicle based on genetic algorithm and quadratic programming. J Power Sources 2014; 248: 416–426.

10.

Chen

Xiong

Wang

, et al. Optimal energy management strategy of a plug-in hybrid electric vehicle based on a particle swarm optimization algorithm. Energies 2015; 8(5): 3661–3678.

11.

Liu

A bi-level control for energy efficiency improvement of a hybrid tracked vehicle. IEEE Trans Industr Inform 2018; 14(4): 1616–1625.

12.

Sezer

Gokasan

Bogosyan

A novel ECMS and combined cost map approach for high-efficiency series hybrid electric vehicles. IEEE Trans Veh Technol 2011; 60(8): 3557–3570.

13.

Hsu

Chen

, et al. A reinforcement learning based dynamic power management for fuel cell hybrid electric vehicle. In: 2016 joint 8th international conference on soft computing and intelligent systems (SCIS) and 17th international symposium on advanced intelligent systems (ISIS), Sapporo, Japan, 25–28 August 2016, pp.460–464. New York: IEEE.

14.

Degris

Pilarski

Sutton

RS.

Model-free reinforcement learning with continuous action in practice. In: 2012 American control conference (ACC), Montreal, QC, Canada, 27–29 June 2012, pp.2177–2182. New York: IEEE.

15.

Montague

PR.

Reinforcement learning: an introduction, by Sutton, RS and Barto, AG. Trends Cogn Sci 1999; 3(9): 360.

16.

Zou

Liu

, et al. Reinforcement learning-based real-time energy management for a hybrid tracked vehicle. Appl Energy 2016; 171: 372–382.

17.

Filipe

Bessa

Reis

, et al. Data-driven predictive energy optimization in a wastewater pumping station. Appl Energy 2019; 252: 113423.

18.

Han

, et al. Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle. Appl Energy 2019; 254: 113708.

19.

Liu

Wang

, et al. Load-adaptive real-time energy management strategy for battery/ultracapacitor hybrid energy storage system using dynamic programming optimization. J Power Sources 2019; 438: 227024.

20.

Zou

Fan

Dong

, et al. DQL energy management: an online-updated algorithm and its application in fix-line hybrid electric vehicle. Energy 2021; 225: 120174.

21.

Huo

Zhang

, et al. Deep deterministic policy gradient based energy management strategy for hybrid electric tracked vehicle with online updating mechanism. IEEE Access 2021; 9: 7280–7292.

22.

Hou

Shi

, et al. Learning time reduction using warm-start methods for a reinforcement learning-based supervisory control in hybrid electric vehicle applications. IEEE Trans Transp Electrif 2020; 7(2): 626–635.

23.

Khajepour

, et al. Energy management for a power-split hybrid electric bus via deep reinforcement learning with terrain information. Appl Energy 2019; 255: 113762.

24.

Tang

Chen

, et al. Double deep reinforcement learning-based energy management for a parallel hybrid electric vehicle with engine start–stop strategy. IEEE Trans Transp Electrif 2021; 8(1): 1376–1388.

25.

Wang

Bai

, et al. Parameterized deep Q-network based energy management with balanced energy economy and battery life for hybrid electric vehicles. Appl Energy 2022; 320: 119270.

26.

Ruan

Wei

, et al. Hierarchical control strategy for the hybrid electric propulsion system of a flying car with engine start-stop system and dynamic coordination. IEEE Trans Transp Electrif 2023; 10(3): 4904–4918.

27.

Peng

, et al. Deep reinforcement learning-based energy management for a series hybrid electric vehicle enabled by history cumulative trip information. IEEE Trans Veh Technol 2019; 68(8): 7416–7430.

28.

Bester

James

Konidaris

GD.

Multi-pass Q-networks for deep reinforcement learning with parameterised action spaces. arXiv preprint arXiv:1905.04388, 2019.

29.

Zhang

Jiao

Yang

A double-deep Q-network-based energy management strategy for hybrid electric vehicles under variable driving cycles. Energy Technol 2021; 9(2): 2000770.

30.

Lian

Tan

Peng

, et al. Cross-type transfer for deep reinforcement learning based hybrid electric vehicle energy management. IEEE Trans Veh Technol 2020; 69(8): 8367–8380.

31.

Zhang

Peng

Tan

, et al. A deep reinforcement learning-based energy management framework with Lagrangian relaxation for plug-in hybrid electric vehicle. IEEE Trans Transp Electrif 2020; 7(3): 1146–1160.