Data-driven characterization of power distribution strategies in hybrid electric vehicles using the actor-critic method

Abstract

The optimization of power distribution in hybrid electric vehicles (HEVs) under unpredictable real-world driving conditions presents a significant challenge. The complexity arises from the myriad transient states that vehicles encounter, along with the necessity to maintain the State of Charge (SoC) amid stringent environmental regulations. This study aims to demonstrate the potential for enhancing power distribution strategies in real-world driving scenarios within HEV systems. To accurately reflect the operating conditions of actual HEVs, we utilized a Simulink model that mirrors the structure and performance of real vehicles. To increase the model’s validity and applicability, we trained it using real driving emissions (RDE) data instead of artificially generated driving cycles. Specifically, we employed the proximal policy optimization algorithm in an actor-critic framework to ensure training stability and data efficiency. Our proposed approach integrates nuanced control by segmenting power distribution into stages, considering engine usage and torque adjustment, thereby making the controller’s actions more descriptive, and interpretable. We demonstrated the effectiveness of this approach by comparing the results with rule-based methods and models trained solely on standardized driving profiles. The proposed method successfully maintained the SoC within a 1.4% deviation from the baseline value while optimizing fuel economy to 14.59 km/L for the Worldwide Harmonized Light Vehicles Test Procedure (WLTP) cycle. This study pioneers a comprehensive framework that combines advanced reinforcement learning techniques with sophisticated vehicle modeling using RDE data, offering a promising solution for optimizing HEV performance in diverse driving conditions.

Keywords

Hybrid electric vehicles power distribution reinforcement learning proximal policy optimization real driving emissions engine usage torque adjustment state of charge fuel efficiency

Get full access to this article

View all access options for this article.

References

Watkins

CJCH

Dayan

. Q-learning. Mach Learn 1992; 8: 279–292.

Zhang

Wang

Liu

, et al. Reinforcement learning-based intelligent energy management architecture for hybrid construction machinery. Appl Energy 2020; 275: 115401.

Mnih

Kavukcuoglu

Silver

, et al. Playing atari with deep reinforcement learning. arXiv preprint arXiv:13125602, 2013.

Zou

Dong

, et al. A self-adaptive energy management strategy for plug-in hybrid electric vehicle based on deep q learning. In: J. Phys. Conf. Ser., vol. 1576, p. 012037. Haugzhou, China: IOP Publishing, 2020.

Han

, et al. Energy management based on reinforcement learning with double deep q-learning for a hybrid electric tracked vehicle. Appl Energy 2019; 254: 113708.

Peng

, et al. Continuous reinforcement learning of energy management with deep q network for a power split hybrid electric bus. Appl Energy 2018; 222: 799–811.

Lillicrap

Hunt

Pritzel

, et al. Continuous control with deep reinforcement learning. arXiv preprint arXiv:150902971, 2015.

Lian

Tan

Peng

, et al. Cross-type transfer for deep reinforcement learning based hybrid electric vehicle energy management. IEEE Trans Vehicular Technol 2020; 69(8): 8367–8380.

Peng

, et al. Deep reinforcement learning-based energy management for a series hybrid electric vehicle enabled by history cumulative trip information. IEEE Trans Vehicular Technol 2019; 68(8): 7416–7430.

10.

Fujimoto

Hoof

Meger

. Addressing function approximation error in actor-critic methods. In: Proceeding International conference on machine learning, Stockholm, Sweden, 2018, pp.1587–1596. PMLR.

11.

Zhou

Xue

, et al. A novel energy management strategy of hybrid electric vehicle via an improved td3 deep reinforcement learning. Energy 2021; 224: 120118.

12.

Haarnoja

Zhou

Abbeel

, et al. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceeding International conference on machine learning, Stockholm, Sweden, pp. 2018. PMLR.

13.

Schulman

Wolski

Dhariwal

, et al. Proximal policy optimization algorithms. arXiv preprint arXiv:170706347, 2017.

14.

Liu

Murphey

YL.

Optimal power management based on q-learning and neuro-dynamic programming for plug-in hybrid electric vehicles. IEEE Trans Neural Netw Learn Syst 2020; 31(6): 1942–1954.

15.

Zhang

Jiao

Yang

A Double-Deep Q-Network-Based Energy Management Strategy for hybrid electric vehicles under variable driving cycles. Energy Technol 2021; 9(2): 2000770.

16.

Zhang

Zou

Zhang

, et al. Online updating energy management strategy based on deep reinforcement learning with accelerated training for hybrid electric tracked vehicles. IEEE Trans Transp Electrification 2022; 8(3): 3289–3306.

17.

Xiao

Yang

, et al. Energy management strategy via maximum entropy reinforcement learning for an extended range logistics vehicle. Energy 2022; 253: 124105.

18.

Zhu

Gupta

, et al. A deep reinforcement learning framework for eco-driving in connected and automated hybrid electric vehicles. IEEE Trans Vehicular Technol 2024; 73: 1713–1725.

19.

Mnih

Kavukcuoglu

Silver

, et al. Human-level control through deep reinforcement learning. Nature 2015; 518(7540): 529–533.

20.

Hyundai Motor Company. System and method for controlling engine and motor of environment-friendly vehicle. 2017. Patent No. KR101714206B1, Korean Intellectual Property Office. https://patents.google.com/patent/KR101714206B1/en.

21.

Lee

Song

Kim

, et al. Comparative analysis of energy management strategies for hev: Dynamic programming and reinforcement learning. IEEE Access 2020; 8: 67112–67123.

22.

Sutton

Barto

AG.

Reinforcement learning: an introduction. Cambridge, MA: MIT Press, 2018.

23.

Konda

Tsitsiklis

Actor-critic algorithms. Adv Neural Inf Process 1999; 12: 1008–1014.

24.

Sutton

RS.

Learning to predict by the methods of temporal differences. Mach Learn 1988; 3: 9–44.

25.

Robert

Casella

Monte Carlo statistical methods, vol. 2. New York, NY: Springer, 1999.