Tackling the Non-IID Issue in Heterogeneous Federated Learning by Gradient Harmonization
This work addresses a key bottleneck in federated learning for privacy-preserving collaborative training, offering an incremental improvement as a plug-and-play module.
The paper tackles the problem of non-IID data and device heterogeneity in federated learning by addressing gradient conflicts, proposing FedGH, a gradient harmonization method that projects gradients onto orthogonal planes to mitigate local drifts. Experiments show FedGH consistently enhances state-of-the-art FL baselines across diverse benchmarks, with more significant improvements in stronger heterogeneity scenarios.
Federated learning (FL) is a privacy-preserving paradigm for collaboratively training a global model from decentralized clients. However, the performance of FL is hindered by non-independent and identically distributed (non-IID) data and device heterogeneity. In this work, we revisit this key challenge through the lens of gradient conflicts on the server side. Specifically, we first investigate the gradient conflict phenomenon among multiple clients and reveal that stronger heterogeneity leads to more severe gradient conflicts. To tackle this issue, we propose FedGH, a simple yet effective method that mitigates local drifts through Gradient Harmonization. This technique projects one gradient vector onto the orthogonal plane of the other within conflicting client pairs. Extensive experiments demonstrate that FedGH consistently enhances multiple state-of-the-art FL baselines across diverse benchmarks and non-IID scenarios. Notably, FedGH yields more significant improvements in scenarios with stronger heterogeneity. As a plug-and-play module, FedGH can be seamlessly integrated into any FL framework without requiring hyperparameter tuning.