Gradient Projection onto Historical Descent Directions for Communication-Efficient Federated Learning
This addresses communication bottlenecks in federated learning for decentralized model training, though it appears incremental as it builds on existing compression techniques.
The paper tackles communication efficiency in federated learning by introducing two algorithms (ProjFL and ProjFL+EF) that project local gradients onto historical descent directions, achieving accuracy comparable to baselines while substantially reducing communication costs.
Federated Learning (FL) enables decentralized model training across multiple clients while optionally preserving data privacy. However, communication efficiency remains a critical bottleneck, particularly for large-scale models. In this work, we introduce two complementary algorithms: ProjFL, designed for unbiased compressors, and ProjFL+EF, tailored for biased compressors through an Error Feedback mechanism. Both methods rely on projecting local gradients onto a shared client-server subspace spanned by historical descent directions, enabling efficient information exchange with minimal communication overhead. We establish convergence guarantees for both algorithms under strongly convex, convex, and non-convex settings. Empirical evaluations on standard FL classification benchmarks with deep neural networks show that ProjFL and ProjFL+EF achieve accuracy comparable to existing baselines while substantially reducing communication costs.