BICompFL: Stochastic Federated Learning with Bi-Directional Compression
This addresses the communication constraints in federated learning for distributed systems, representing an incremental improvement with specific gains.
The paper tackles the communication bottleneck in federated learning by proposing BICompFL, a method for stochastic federated learning with bi-directional compression, which reduces communication cost by an order of magnitude while maintaining state-of-the-art accuracies.
We address the prominent communication bottleneck in federated learning (FL). We specifically consider stochastic FL, in which models or compressed model updates are specified by distributions rather than deterministic parameters. Stochastic FL offers a principled approach to compression, and has been shown to reduce the communication load under perfect downlink transmission from the federator to the clients. However, in practice, both the uplink and downlink communications are constrained. We show that bi-directional compression for stochastic FL has inherent challenges, which we address by introducing BICompFL. Our BICompFL is experimentally shown to reduce the communication cost by an order of magnitude compared to multiple benchmarks, while maintaining state-of-the-art accuracies. Theoretically, we study the communication cost of BICompFL through a new analysis of an importance-sampling based technique, which exposes the interplay between uplink and downlink communication costs.