Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEs
This work addresses communication efficiency for distributed machine learning systems, particularly in noisy environments like language modeling, but is incremental as it compares existing compression methods.
The paper tackled the problem of communication bottlenecks in distributed learning by analyzing the noise resilience of compressed SGD methods, finding that Distributed SignSGD remains robust to large and heavy-tailed gradient noise while unbiased compression methods are more vulnerable, with empirical validation across architectures and datasets.
Distributed methods are essential for handling machine learning pipelines comprising large-scale models and datasets. However, their benefits often come at the cost of increased communication overhead between the central server and agents, which can become the main bottleneck, making training costly or even unfeasible in such systems. Compression methods such as quantization and sparsification can alleviate this issue. Still, their robustness to large and heavy-tailed gradient noise, a phenomenon sometimes observed in language modeling, remains poorly understood. This work addresses this gap by analyzing Distributed Compressed SGD (DCSGD) and Distributed SignSGD (DSignSGD) using stochastic differential equations (SDEs). Our results show that DCSGD with unbiased compression is more vulnerable to noise in stochastic gradients, while DSignSGD remains robust, even under large and heavy-tailed noise. Additionally, we propose new scaling rules for hyperparameter tuning to mitigate performance degradation due to compression. These findings are empirically validated across multiple deep learning architectures and datasets, providing practical recommendations for distributed optimization.