Collaborative scheduling optimization of hydrogen-enhanced integrated energy system via goal-conditioned hierarchical reinforcement learning

Under carbon peak and carbon neutrality targets, energy systems are evolving toward diversified architectures that integrate various energy forms. However, the involvement of multiple energy carriers poses challenges to coordinated optimal scheduling, including temporal heterogeneity, a contracted feasible decision space, and a trade-off between volatility and robustness. To overcome these challenges, a hydrogen-enhanced integrated energy system (H-IES) is introduced, in which a hydrogen storage system (HSS) and a battery energy storage system (BESS) jointly coordinate scheduling to balance short-term load response and long-term energy transfers. To formalize the H-IES scheduling task, two augmented Markov decision process (MDP) models are established, with a semi-MDP (SMDP) for high-level planning and a goal-augmented MDP (GAMDP) for low-level sequential control, mitigating the “short-sightedness” of single time-scale models. To address the above models, a goal-conditioned hierarchical reinforcement learning approach (GHTD3) with twin delayed deep deterministic policy gradient as internal controllers is proposed, to achieve coordinated energy allocation through high-level long-horizon planning and low-level short-horizon response. To improve the stability of algorithm, a historical goal relabeling mechanism is incorporated to correct distributional drift in high-level replayed experiences. Simulations across three seasonal scenarios reveal that, compared with mathematical programming method and heuristic algorithm, reinforcement learning demonstrates a stronger ability to exploit the value of the hydrogen pathway and to learn stable energy storage dispatch strategies. Compared with the single-level TD3, GHTD3 delivers higher cumulative rewards and, by flexibly staggering fuel cell (FC) operation during cooling or heat peaks, reduces cost by 22.6% and cuts renewable energy curtailment by 85.9%. These findings confirm GHTD3’s superior coordination, robustness, and cost-effectiveness in multi-energy scheduling.