Multi-objective federated learning: Balancing global performance and individual fairness

In federated learning, non-iid data not only diminishes the performance of the global model but also gives rise to the fairness problem which manifests as an increase in the variance of the global model's accuracy across clients. Fairness issues can result in the global model performing poorly or even failing on certain clients. Existing methods addressing the fairness problem in federated learning tend to neglect the comprehensive improvement of both the average performance and fairness of the global model. In addressing it, the multi-objective optimization method for fine-tuning global gradients, FedMC algorithm is introduced in this paper. The primary objective is the average loss function of all clients, and the sub-objective involves fine-tuning the global gradient by reducing the gradient conflict between the global gradient and the local gradients. Specifically, we refine the global gradient by incorporating a sub-optimization objective aimed at alleviating conflicts between the global gradient and the local gradient with the largest deviation, denoted as FedMC. FedMC can enhance the performance and convergence rate of clients with initially poor performance, albeit at the cost of the earlier convergence rate of clients with initially good performance. Nevertheless, it enables the latter to reach the accuracy level achieved before fine-tuning. In addition, we also propose FedMC+ algorithm, owning three additional optimization mechanisms built upon the FedMC optimization objective which includes the decay of hyperparameter, the sliding window mechanism, and data-balanced client selection. Besides, we present a theoretical analysis of the convergence rate of FedMC, demonstrating its convergence to a Pareto stationary solution. Our combined experimental results confirm that FedMC+ achieves an average 4.5% improvement in accuracy and a 22% reduction in the degree of dispersion compared to state-of-the-art federated learning (FL) methods.