Communication Compression for Distributed Learning without Control Variates
This addresses a key bottleneck in federated learning by enabling efficient communication compression without compromising privacy or requiring stateful clients, though it is incremental as it builds on existing distributed gradient descent methods.
The paper tackles the problem of biased compression in distributed learning, which requires error feedback with client-specific control variates that violate privacy and demand stateful clients, by proposing CAFe, a framework that uses past aggregated updates to allow highly compressible client updates without control variates, and it outperforms existing schemes in experiments.
Distributed learning algorithms, such as the ones employed in Federated Learning (FL), require communication compression to reduce the cost of client uploads. The compression methods used in practice are often biased, making error feedback necessary both to achieve convergence under aggressive compression and to provide theoretical convergence guarantees. However, error feedback requires client-specific control variates, creating two key challenges: it violates privacy-preserving principles and demands stateful clients. In this paper, we propose Compressed Aggregate Feedback (CAFe), a novel distributed learning framework that allows highly compressible client updates by exploiting past aggregated updates, and does not require control variates. We consider Distributed Gradient Descent (DGD) as a representative algorithm and analytically prove CAFe's superiority to Distributed Compressed Gradient Descent (DCGD) with biased compression in the non-convex regime with bounded gradient dissimilarity. Experimental results confirm that CAFe outperforms existing distributed learning compression schemes.