Energy management strategy for hybrid electric propulsion system based on parametric deep Q-network in hybrid action space

Abstract

This study proposes an energy management strategy (EMS) for hybrid power systems based on model-free deep reinforcement learning using parametric Deep Q-Network (P-DQN/MP-DQN). The traditional deep reinforcement learning (DRL) paradigm-based EMS is limited by its self-learning capabilities, making it unable to simultaneously achieve hybrid action control of shifting rules and overall vehicle EMS. The EMS using DDPG (Deep Deterministic Policy Gradient) with a separately designed single-parameter shifting rule achieves about 94% of the fuel economy of single-parameter DP (Dynamic Programming). There remains significant room for improvement in DRL-based EMS that includes transmission shifting strategies. To address this, we employed a parametric action space to enhance hybrid action control in the EMS and used a Multi-Pass Q-network (MP-QN) to decouple the coherence of Q-values for discrete actions. The multi-channel architecture of MP-QN enables each channel to independently learn and optimize the Q value of a specific action, enhancing the ability to handle complex decision-making processes. The results show that the EMS based on P-DQN and MP-DQN formulated an excellent intelligent shifting rule similar to multi-parameter DP (DP-m) and significantly improved global optimality. Compared to the EMS based on single-parameter shifting rule DP (DP-s), fuel economy is improved. Additionally, the MP-DQN using the Multi-Pass Q-network significantly improved the stability and fuel economy of EMS training convergence, demonstrating its excellent potential for hybrid action control.