FedBCD:Communication-Efficient Accelerated Block Coordinate Gradient Descent for Federated Learning
This work provides a method to significantly reduce communication overhead in federated learning, which is a critical bottleneck for training large-scale deep models.
This paper addresses the high communication overhead in Federated Learning for large models by proposing FedBCGD, which splits model parameters into blocks and allows clients to upload only specific blocks. The accelerated version, FedBCGD+, further incorporates client drift control and stochastic variance reduction, achieving a communication complexity that is a factor of 1/N lower than existing methods, where N is the number of parameter blocks.
Although Federated Learning has been widely studied in recent years, there are still high overhead expenses in each communication round for large-scale models such as Vision Transformer. To lower the communication complexity, we propose a novel Federated Block Coordinate Gradient Descent (FedBCGD) method for communication efficiency. The proposed method splits model parameters into several blocks, including a shared block and enables uploading a specific parameter block by each client, which can significantly reduce communication overhead. Moreover, we also develop an accelerated FedBCGD algorithm (called FedBCGD+) with client drift control and stochastic variance reduction. To the best of our knowledge, this paper is the first work on parameter block communication for training large-scale deep models. We also provide the convergence analysis for the proposed algorithms. Our theoretical results show that the communication complexities of our algorithms are a factor $1/N$ lower than those of existing methods, where $N$ is the number of parameter blocks, and they enjoy much faster convergence than their counterparts. Empirical results indicate the superiority of the proposed algorithms compared to state-of-the-art algorithms. The code is available at https://github.com/junkangLiu0/FedBCGD.