Communication-efficient Federated Learning with Single-Step Synthetic Features Compressor for Faster Convergence
This addresses communication bottlenecks in large-scale distributed privacy-preserving machine learning, offering an incremental improvement over existing compression methods.
The paper tackles the problem of high communication overhead in federated learning by proposing a single-step synthetic features compressor (3SFC) that constructs a tiny synthetic dataset from raw gradients, achieving a compression rate as low as 0.02% and significantly better convergence rates compared to competing methods.
Reducing communication overhead in federated learning (FL) is challenging but crucial for large-scale distributed privacy-preserving machine learning. While methods utilizing sparsification or others can largely lower the communication overhead, the convergence rate is also greatly compromised. In this paper, we propose a novel method, named single-step synthetic features compressor (3SFC), to achieve communication-efficient FL by directly constructing a tiny synthetic dataset based on raw gradients. Thus, 3SFC can achieve an extremely low compression rate when the constructed dataset contains only one data sample. Moreover, 3SFC's compressing phase utilizes a similarity-based objective function so that it can be optimized with just one step, thereby considerably improving its performance and robustness. In addition, to minimize the compressing error, error feedback (EF) is also incorporated into 3SFC. Experiments on multiple datasets and models suggest that 3SFC owns significantly better convergence rates compared to competing methods with lower compression rates (up to 0.02%). Furthermore, ablation studies and visualizations show that 3SFC can carry more information than competing methods for every communication round, further validating its effectiveness.