Communication-Efficient Federated Learning with Dual-Side Low-Rank Compression
This addresses communication bottlenecks in federated learning for distributed clients, but it is incremental as it builds on existing compression techniques.
The paper tackles the communication-intensive training challenge in federated learning by proposing FedDLR, a method using dual-side low-rank compression, which reduces communication overhead and speeds up inference, with empirical results showing it outperforms state-of-the-art solutions in communication and computation efficiency.
Federated learning (FL) is a promising and powerful approach for training deep learning models without sharing the raw data of clients. During the training process of FL, the central server and distributed clients need to exchange a vast amount of model information periodically. To address the challenge of communication-intensive training, we propose a new training method, referred to as federated learning with dual-side low-rank compression (FedDLR), where the deep learning model is compressed via low-rank approximations at both the server and client sides. The proposed FedDLR not only reduces the communication overhead during the training stage but also directly generates a compact model to speed up the inference process. We shall provide convergence analysis, investigate the influence of the key parameters, and empirically show that FedDLR outperforms the state-of-the-art solutions in terms of both the communication and computation efficiency.