Fed-GraB: Federated Long-tailed Learning with Self-Adjusting Gradient Balancer
This addresses data privacy and class imbalance in federated learning, which is crucial for real-world applications like medical imaging or autonomous driving, though it is incremental as it builds on existing federated and long-tailed learning methods.
The paper tackles federated long-tailed learning, where clients have heterogeneous data that collectively form a long-tailed distribution, by proposing Fed-GraB with a self-adjusting gradient balancer to address distribution drift and improve performance on minority classes. It achieves state-of-the-art results on datasets like CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist.
Data privacy and long-tailed distribution are the norms rather than the exception in many real-world tasks. This paper investigates a federated long-tailed learning (Fed-LT) task in which each client holds a locally heterogeneous dataset; if the datasets can be globally aggregated, they jointly exhibit a long-tailed distribution. Under such a setting, existing federated optimization and/or centralized long-tailed learning methods hardly apply due to challenges in (a) characterizing the global long-tailed distribution under privacy constraints and (b) adjusting the local learning strategy to cope with the head-tail imbalance. In response, we propose a method termed $\texttt{Fed-GraB}$, comprised of a Self-adjusting Gradient Balancer (SGB) module that re-weights clients' gradients in a closed-loop manner, based on the feedback of global long-tailed distribution evaluated by a Direct Prior Analyzer (DPA) module. Using $\texttt{Fed-GraB}$, clients can effectively alleviate the distribution drift caused by data heterogeneity during the model training process and obtain a global model with better performance on the minority classes while maintaining the performance of the majority classes. Extensive experiments demonstrate that $\texttt{Fed-GraB}$ achieves state-of-the-art performance on representative datasets such as CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist.