Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization
It addresses communication efficiency for scalable distributed and federated learning systems, though it appears incremental as it builds on existing paradigms with hybrid improvements.
This dissertation tackles communication bottlenecks in distributed and federated learning by developing strategies like compression, local training, and personalization, achieving superior performance under IID and non-IID settings with frameworks like Scafflix and SymWanda that maintain accuracy while reducing communication costs.
Distributed and federated learning are essential paradigms for training models across decentralized data sources while preserving privacy, yet communication overhead remains a major bottleneck. This dissertation explores strategies to improve communication efficiency, focusing on model compression, local training, and personalization. We establish a unified framework for biased and unbiased compression operators with convergence guarantees, then propose adaptive local training strategies that incorporate personalization to accelerate convergence and mitigate client drift. In particular, Scafflix balances global and personalized objectives, achieving superior performance under both IID and non-IID settings. We further introduce privacy-preserving pruning frameworks that optimize sparsity while minimizing communication costs, with Cohort-Squeeze leveraging hierarchical aggregation to reduce cross-device overhead. Finally, SymWanda, a symmetric post-training pruning method, enhances robustness under high sparsity and maintains accuracy without retraining. Extensive experiments on benchmarks and large-scale language models demonstrate favorable trade-offs among accuracy, convergence, and communication, offering theoretical and practical insights for scalable, efficient distributed learning.