With the rapid development of automotive intelligence, energy management strategies (EMS) for hybrid electric buses (HEBs) face higher demands. Traditional EMS cannot effectively handle dynamic changes in operational states, such as time-varying vehicle mass and road slope. To address this, this study develops a hierarchical EMS that accounts for uncertain operational states and proposes a novel EMS optimization algorithm based on deep reinforcement learning (DRL) for torque distribution in HEBs. First, to improve the adaptability of the EMS to various regions and road conditions, Extended Kalman Filtering (EKF) is used to jointly estimate real-time vehicle mass and road slope. Combined with historical operating data, Dynamic Programming (DP) performs global planning of operating conditions. The optimal experiences are integrated into experience replay and considered in reward function. Second, to tackle the sparse reward issue during training, the Twin Delayed Deep Deterministic Policy Gradient (TD3) framework is enhanced with annealing bias-priority experience replay and sample learning rate annealing, improving training efficiency. Finally, the proposed strategy is validated in a synthesized driving environment for fixed-route bus operations. Validation results show that, compared to traditional TD3-based EMS, the proposed method improves learning capability by 51.0%, convergence speed by 13.1%, and fuel economy by 9.4%.
HardmanS.Understanding the impact of reoccurring and non-financial incentives on plug-in electric vehicle adoption - a review. Transp Res Part A Policy Pract2019; 119: 1–14.
2.
LebkowskiA.Studies of energy consumption by a city bus powered by a hybrid energy storage system in variable road conditions. Energies2019; 12(5): 951.
3.
XieSHuXXinZ, et al. Pontryagin’s Minimum Principle based model predictive control of energy management for a plug-in hybrid electric bus. Appl Energy2019; 236: 893–905.
4.
DongPZhaoJLiuX, et al. Practical application of energy management strategy for hybrid electric vehicles based on intelligent and connected technologies: development stages, challenges, and future trends. Renew Sustain Energy Rev2022; 170: 112947.
5.
LuDHuDYiF, et al. Optimal selection range of FCV power battery capacity considering the synergistic decay of dual power source lifespan. Int J Hydrogen Energy2023; 48: 13578–13590.
6.
ZhaoXWangLZhouY, et al. Energy management strategies for fuel cell hybrid electric vehicles: classification, comparison, and outlook. Energy Convers Manag2022; 270: 116179.
7.
LiuYLiuJZhangY, et al. Rule learning based energy management strategy of fuel cell hybrid vehicles considering multi-objective optimization. Energy2020; 207: 118212.
8.
TengTZhangXDongH, et al. A comprehensive review of energy management optimization strategies for fuel cell passenger vehicle. Int J Hydrogen Energy2020; 45: 20293–20303.
9.
LiuJChenYZhanJ, et al. Heuristic dynamic programming based online energy management strategy for plug-in hybrid electric vehicles. IEEE Trans Veh Technol2019; 68: 4479–4493.
10.
LiLYangCZhangY, et al. Correctional DP-based energy management strategy of plug-in hybrid electric bus for city-bus route. IEEE Trans Veh Technol2015; 64: 2792–2803.
11.
LeeWJeoungHParkD, et al. An adaptive concept of PMP-based control for saving operating costs of extended-range electric vehicles. IEEE Trans Veh Technol2019; 68: 11505–11512.
12.
HeHJiaCLiJ.A new cost-minimizing power-allocating strategy for the hybrid electric bus with fuel cell/battery health-aware control. Int J Hydrogen Energy2022; 47: 22147–22164.
13.
GaoKLuoPXieJ, et al. Energy management of plug-in hybrid electric vehicles based on speed prediction fused driving intention and LIDAR. Energy2023; 284: 128535.
14.
CuiWCuiNLiT, et al. An efficient multi-objective hierarchical energy management strategy for plug-in hybrid electric vehicle in connected scenario. Energy2022; 257: 124690.
15.
HuoWChenDTianS, et al. Lifespan-consciousness and minimum-consumption coupled energy management strategy for fuel cell hybrid vehicles via deep reinforcement learning. Int J Hydrogen Energy2022; 47: 24026–24041.
16.
WuJHuangZHuangW, et al. Prioritized experience-based reinforcement learning with human guidance for autonomous driving. IEEE Trans Neural Netw Learn Syst2024; 35: 855–869.
17.
ZhengCZhangDXiaoY, et al. Reinforcement learning-based energy management strategies of fuel cell hybrid vehicles with multi-objective control. J Power Sources2022; 543: 231841.
18.
XuBRathodDZhangD, et al. Parametric study on reinforcement learning optimized energy management strategy for a hybrid electric vehicle. Appl Energy2020; 259: 11420.
19.
XuBShiJLiS, et al. Energy consumption and battery aging minimization using a Q-learning strategy for a battery/ultracapacitor electric vehicle. Energy2021; 229: 120705.
20.
WuPPartridgeJAnderliniE, et al. Near-optimal energy management for plug-in hybrid fuel cell and battery propulsion using deep reinforcement learning. Int J Hydrogen Energy2021; 46: 40022–40040.
21.
ZhouJXueSXueY, et al. A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning. Energy2021; 224: 120118.
22.
SilverDHuangAMaddisonJ, et al. Mastering the game of Go with deep neural networks and tree search. Nature2016; 529(7587): 484–489.
23.
WuJHeHPengJ, et al. Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus. Appl Energy2018; 222: 799–811.
24.
TangXZhouHWangF, et al. Longevity-conscious energy management strategy of fuel cell hybrid electric vehicle based on deep reinforcement learning. Energy2022; 238: 121593.
25.
GuoJLiWWangB, et al. Deep reinforcement learning-based control strategy of connected hybrid electric vehicles platooning. J Mech Eng2024; 60: 262–271.
26.
LiKJiaCHanX, et al. A novel minimal-cost power allocation strategy for fuel cell hybrid buses based on deep reinforcement learning algorithms. Sustainability2023; 15: 7967.
27.
CuiNCuiWShiY.Deep reinforcement learning based PHEV energy management with co-recognition for traffic condition and driving style. IEEE Trans Intell Veh2023; 8: 3026–3039.
28.
WuYTanHPengJ, et al. Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus. Appl Energy2019; 247: 454–466.
29.
LianRPengJWuY, et al. Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle. Energy2020; 197: 117297.
30.
RuanJWuCLiangZ, et al. The application of machine learning-based energy management strategy in a multi-mode plug-in hybrid electric vehicle, part II: deep deterministic policy gradient algorithm design for electric mode. Energy2023; 269: 126792.
31.
HuangRHeH.A novel data-driven energy management strategy for fuel cell hybrid electric bus based on improved twin delayed deep deterministic policy gradient algorithm. Int J Hydrogen Energy2024; 52: 782–798.
32.
JiaCZhouJHeH, et al. Health-conscious deep reinforcement learning energy management for fuel cell buses integrating environmental and look-ahead road information. Energy2024; 290: 130146.
33.
SoldoJSkugorBDeurJ.Analysis of optimal battery state-of-charge trajectory patterns for blended mode of a parallel plug-in hybrid electric vehicle and a wide range of driving conditions. Optim Eng2021; 22: 1955–1977.
34.
CuiWCuiNLiT, et al. Multi-objective hierarchical energy management for connected plug-in hybrid electric vehicle with cyber-physical interaction. Appl Energy2024; 360: 122816.
35.
WangJDuCYanF, et al. Hierarchical rewarding deep deterministic policy gradient strategy for energy management of hybrid electric vehicles. IEEE Trans Transp Electrif2024; 10: 1802–1815.
36.
HuangRHeHSuQ.Smart energy management for hybrid electric bus via improved soft actor-critic algorithm in a heuristic learning framework. Energy2024; 309: 133091.
37.
JiaCLiKHeH, et al. Health-aware energy management strategy for fuel cell hybrid bus considering air-conditioning control based on TD3 algorithm. Energy2023; 283: 128462.
38.
WangXLuoYQinB, et al. Power allocation strategy for urban rail HESS based on deep reinforcement learning sequential decision optimization. IEEE Trans Transp Electrif2023; 9: 2693–2710.
