Optimal Rate Adaption in Federated Learning with Compressed Communications
This work addresses communication overhead in Federated Learning, which is a bottleneck for distributed machine learning systems, by providing a systematic approach to optimize compression rates, though it is incremental as it builds on existing compression methods.
The paper tackles the tradeoff between compression and model accuracy in Federated Learning by analyzing how compression error affects convergence and proposing an adaptation framework to adjust compression rates per iteration, achieving reduced network traffic while maintaining high accuracy on MNIST and CIFAR-10 datasets.
Federated Learning (FL) incurs high communication overhead, which can be greatly alleviated by compression for model updates. Yet the tradeoff between compression and model accuracy in the networked environment remains unclear and, for simplicity, most implementations adopt a fixed compression rate only. In this paper, we for the first time systematically examine this tradeoff, identifying the influence of the compression error on the final model accuracy with respect to the learning rate. Specifically, we factor the compression error of each global iteration into the convergence rate analysis under both strongly convex and non-convex loss functions. We then present an adaptation framework to maximize the final model accuracy by strategically adjusting the compression rate in each iteration. We have discussed the key implementation issues of our framework in practical networks with representative compression algorithms. Experiments over the popular MNIST and CIFAR-10 datasets confirm that our solution effectively reduces network traffic yet maintains high model accuracy in FL.