SPATL: Salient Parameter Aggregation and Transfer Learning for Heterogeneous Clients in Federated Learning
This addresses efficiency and stability issues for edge devices in federated learning, though it appears incremental.
The paper tackled communication costs and data heterogeneity in federated learning by proposing SPATL, which reduces communication overhead by up to 86.45% and accelerates local inference by up to 39.7% FLOPs on VGG-11.
Federated learning~(FL) facilitates the training and deploying AI models on edge devices. Preserving user data privacy in FL introduces several challenges, including expensive communication costs, limited resources, and data heterogeneity. In this paper, we propose SPATL, an FL method that addresses these issues by: (a) introducing a salient parameter selection agent and communicating selected parameters only; (b) splitting a model into a shared encoder and a local predictor, and transferring its knowledge to heterogeneous clients via the locally customized predictor. Additionally, we leverage a gradient control mechanism to further speed up model convergence and increase robustness of training processes. Experiments demonstrate that SPATL reduces communication overhead, accelerates model inference, and enables stable training processes with better results compared to state-of-the-art methods. Our approach reduces communication cost by up to $86.45\%$, accelerates local inference by reducing up to $39.7\%$ FLOPs on VGG-11, and requires $7.4 \times$ less communication overhead when training ResNet-20.