Decoupled Federated Learning on Long-Tailed and Non-IID data with Feature Statistics
This addresses a specific bottleneck in federated learning for scenarios with heterogeneous data distributions, offering improvements in model performance and convergence.
The paper tackles the problem of federated learning on long-tailed and non-IID data, where tail classes are sparsely distributed, leading to slower convergence and poorer performance. The proposed Decoupled Federated Learning with Feature Statistics (DFL-FS) method outperforms state-of-the-art methods in accuracy and convergence rate on CIFAR10-LT and CIFAR100-LT datasets.
Federated learning is designed to enhance data security and privacy, but faces challenges when dealing with heterogeneous data in long-tailed and non-IID distributions. This paper explores an overlooked scenario where tail classes are sparsely distributed over a few clients, causing the models trained with these classes to have a lower probability of being selected during client aggregation, leading to slower convergence rates and poorer model performance. To address this issue, we propose a two-stage Decoupled Federated learning framework using Feature Statistics (DFL-FS). In the first stage, the server estimates the client's class coverage distributions through masked local feature statistics clustering to select models for aggregation to accelerate convergence and enhance feature learning without privacy leakage. In the second stage, DFL-FS employs federated feature regeneration based on global feature statistics and utilizes resampling and weighted covariance to calibrate the global classifier to enhance the model's adaptability to long-tailed data distributions. We conducted experiments on CIFAR10-LT and CIFAR100-LT datasets with various long-tailed rates. The results demonstrate that our method outperforms state-of-the-art methods in both accuracy and convergence rate.