FedDQ: Communication-Efficient Federated Learning with Descending Quantization
This work addresses communication efficiency for federated learning systems, offering an incremental improvement over prior adaptive quantization schemes.
The paper tackles the communication bottleneck in federated learning by proposing FedDQ, a descending quantization scheme that reduces quantization levels as training progresses, saving up to 65.2% of communicated bit volume and 68% of communication rounds compared to existing methods.
Federated learning (FL) is an emerging learning paradigm without violating users' privacy. However, large model size and frequent model aggregation cause serious communication bottleneck for FL. To reduce the communication volume, techniques such as model compression and quantization have been proposed. Besides the fixed-bit quantization, existing adaptive quantization schemes use ascending-trend quantization, where the quantization level increases with the training stages. In this paper, we first investigate the impact of quantization on model convergence, and show that the optimal quantization level is directly related to the range of the model updates. Given the model is supposed to converge with the progress of the training, the range of the model updates will gradually shrink, indicating that the quantization level should decrease with the training stages. Based on the theoretical analysis, a descending quantization scheme named FedDQ is proposed. Experimental results show that the proposed descending quantization scheme can save up to 65.2% of the communicated bit volume and up to 68% of the communication rounds, when compared with existing schemes.