Addressing the Collaboration Dilemma in Low-Data Federated Learning via Transient Sparsity
This work addresses collaboration inefficiencies in federated learning for privacy-sensitive applications, but it is incremental as it builds on existing FL methods with a specific technique.
The paper tackles the problem of ineffective collaboration in low-data federated learning due to data heterogeneity, identifying a Layer-wise Inertia Phenomenon where middle layers stop updating early, and proposes LIPS, a method using transient sparsity to stimulate updates, which improves performance in various FL scenarios.
Federated learning (FL) enables collaborative model training across decentralized clients while preserving data privacy, leveraging aggregated updates to build robust global models. However, this training paradigm faces significant challenges due to data heterogeneity and limited local datasets, which often impede effective collaboration. In such scenarios, we identify the Layer-wise Inertia Phenomenon in FL, wherein the middle layers of global model undergo minimal updates after early communication rounds, ultimately limiting the effectiveness of global aggregation. We demonstrate the presence of this phenomenon across a wide range of federated settings, spanning diverse datasets and architectures. To address this issue, we propose LIPS (Layer-wise Inertia Phenomenon with Sparsity), a simple yet effective method that periodically introduces transient sparsity to stimulate meaningful updates and empower global aggregation. Experiments demonstrate that LIPS effectively mitigates layer-wise inertia, enhances aggregation effectiveness, and improves overall performance in various FL scenarios. This work not only deepens the understanding of layer-wise learning dynamics in FL but also paves the way for more effective collaboration strategies in resource-constrained environments. Our code is publicly available at: https://github.com/QiaoXiao7282/LIPS.