Heterogeneity-aware Task Scheduling based on Personalized Federated Reinforcement Learning

The workload data generated in large-scale cloud environments is becoming increasingly complex, making collaborative training a promising approach for developing more efficient task schedulers. Considering privacy security and transfer costs, Federated Reinforcement Learning (FRL) emerges as a promising solution. However, our exploratory experiments demonstrate that the environmental heterogeneity contributes to performance degradation in FRL, which makes the above issue challenging. Accordingly, we propose a Personalized FRL method based on Dual-critic networks and Multi-head attention aggregator (PFRL-DM), which achieves the optimal scheduling policies by collaborative training on diverse workload data in heterogeneous environments without exposing private data. We initially introduce a novel Reinforcement Learning (RL) environment modeling, serving as a foundation for the collaborative training of the cloud scheduling agents. Then, we implement a dual-critic network Proximal Policy Optimization (PPO) algorithm for each client, effectively balancing the influence between global and local models on the agents. Furthermore, we integrate multi-head attention weights into the server-side aggregator to implement personalization. Extensive experiments on various real-world workloads have demonstrated that, compared to state-of-the-art algorithm MFPO, the proposed algorithm exhibits faster convergence, shorter response and completion times, and achieves the highest resource utilization. Additionally, the PFRL-DM algorithm constructs personalized models for each client, enabling greater adaptability in heterogeneous and hybrid workload environments. The codes for this paper can be found at https://github.com/liyan2015/PFRL-DM.