Compression with Exact Error Distribution for Federated Learning
This work addresses communication efficiency for Federated Learning systems, offering incremental improvements by refining compression techniques with specific error distributions.
The paper tackles the problem of communication cost in Federated Learning by proposing compression schemes that produce exact error distributions like Gaussian or Laplace, enabling compression-for-free in differential privacy applications and improving standard methods such as Langevin dynamics and randomized smoothing.
Compression schemes have been extensively used in Federated Learning (FL) to reduce the communication cost of distributed learning. While most approaches rely on a bounded variance assumption of the noise produced by the compressor, this paper investigates the use of compression and aggregation schemes that produce a specific error distribution, e.g., Gaussian or Laplace, on the aggregated data. We present and analyze different aggregation schemes based on layered quantizers achieving exact error distribution. We provide different methods to leverage the proposed compression schemes to obtain compression-for-free in differential privacy applications. Our general compression methods can recover and improve standard FL schemes with Gaussian perturbations such as Langevin dynamics and randomized smoothing.