Improved Quantization Strategies for Managing Heavy-tailed Gradients in Distributed Learning
This work solves the problem of communication bottlenecks in distributed learning for practitioners by offering an incremental improvement over existing quantization strategies.
The paper tackles the problem of communication efficiency in distributed deep learning by addressing heavy-tailed gradient distributions that degrade existing quantization methods, resulting in a novel compression scheme that combines gradient truncation with quantization to minimize quantization error and improve performance in distributed SGD.
Gradient compression has surfaced as a key technique to address the challenge of communication efficiency in distributed learning. In distributed deep learning, however, it is observed that gradient distributions are heavy-tailed, with outliers significantly influencing the design of compression strategies. Existing parameter quantization methods experience performance degradation when this heavy-tailed feature is ignored. In this paper, we introduce a novel compression scheme specifically engineered for heavy-tailed gradients, which effectively combines gradient truncation with quantization. This scheme is adeptly implemented within a communication-limited distributed Stochastic Gradient Descent (SGD) framework. We consider a general family of heavy-tail gradients that follow a power-law distribution, we aim to minimize the error resulting from quantization, thereby determining optimal values for two critical parameters: the truncation threshold and the quantization density. We provide a theoretical analysis on the convergence error bound under both uniform and non-uniform quantization scenarios. Comparative experiments with other benchmarks demonstrate the effectiveness of our proposed method in managing the heavy-tailed gradients in a distributed learning environment.