Efficient Distributed Optimization under Heavy-Tailed Noise
This work addresses a critical bottleneck in distributed training for large-scale models, offering an efficient solution to improve training stability and performance under heavy-tailed noise conditions.
The paper tackles the challenge of heavy-tailed stochastic gradient noise in distributed optimization, particularly for attention-based models, by proposing the TailOPT framework with a variant called Bi^2Clip that uses coordinate-wise clipping to achieve adaptive-like performance without extra communication costs, and it demonstrates superior results on language tasks compared to state-of-the-art methods.
Distributed optimization has become the default training paradigm in modern machine learning due to the growing scale of models and datasets. To mitigate communication overhead, local updates are often applied before global aggregation, resulting in a nested optimization approach with inner and outer steps. However, heavy-tailed stochastic gradient noise remains a significant challenge, particularly in attention-based models, hindering effective training. In this work, we propose TailOPT, an efficient framework designed to address heavy-tailed noise by leveraging adaptive optimization or clipping techniques. We establish convergence guarantees for the TailOPT framework under heavy-tailed noise with potentially unbounded gradient variance and local updates. Among its variants, we highlight a memory and communication efficient instantiation which we call $Bi^2Clip$, which performs coordinate-wise clipping at both the inner and outer optimizers, achieving adaptive-like performance (e.g., Adam) without the cost of maintaining or transmitting additional gradient statistics. Empirically, TailOPT, including $Bi^2Clip$, demonstrates superior performance on several language tasks and models, outperforming state-of-the-art methods.