SPARK: Igniting Communication-Efficient Decentralized Learning via Stage-wise Projected NTK and Accelerated Regularization
This work addresses communication efficiency for decentralized learning in bandwidth-limited edge environments, representing a strong specific gain rather than a foundational breakthrough.
The paper tackles the challenges of communication overhead and statistical heterogeneity in decentralized federated learning by proposing SPARK, which integrates random projection-based Jacobian compression, stage-wise annealed distillation, and Nesterov momentum acceleration. It achieves a 98.7% communication reduction compared to NTK-DFL while maintaining convergence speed and superior accuracy, with momentum enabling 3 times faster performance.
Decentralized federated learning (DFL) faces critical challenges from statistical heterogeneity and communication overhead. While NTK-based methods achieve faster convergence, transmitting full Jacobian matrices is impractical for bandwidth-constrained edge networks. We propose SPARK, synergistically integrating random projection-based Jacobian compression, stage-wise annealed distillation, and Nesterov momentum acceleration. Random projections compress Jacobians while preserving spectral properties essential for convergence. Stage-wise annealed distillation transitions from pure NTK evolution to neighbor-regularized learning, counteracting compression noise. Nesterov momentum accelerates convergence through stable accumulation enabled by distillation smoothing. SPARK achieves 98.7% communication reduction compared to NTK-DFL while maintaining convergence speed and superior accuracy. With momentum, SPARK reaches target performance 3 times faster, establishing state-of-the-art results for communication-efficient decentralized learning and enabling practical deployment in bandwidth-limited edge environments.