Revisiting Gradient Normalization and Clipping for Nonconvex SGD under Heavy-Tailed Noise: Necessity, Sufficiency, and Acceleration
This addresses optimization challenges in nonconvex settings with heavy-tailed noise, offering theoretical insights and practical guidance for machine learning practitioners, though it appears incremental as it revisits and extends existing techniques.
The paper challenges the necessity of gradient clipping for SGD convergence under heavy-tailed noise, proving that gradient normalization alone is sufficient for nonconvex SGD and that combining normalization with clipping yields better convergence rates. It provides a unifying theory covering normalization-only, clipping-only, and combined approaches, with an accelerated variant improving convergence under second-order smoothness.
Gradient clipping has long been considered essential for ensuring the convergence of Stochastic Gradient Descent (SGD) in the presence of heavy-tailed gradient noise. In this paper, we revisit this belief and explore whether gradient normalization can serve as an effective alternative or complement. We prove that, under individual smoothness assumptions, gradient normalization alone is sufficient to guarantee convergence of the nonconvex SGD. Moreover, when combined with clipping, it yields far better rates of convergence under more challenging noise distributions. We provide a unifying theory describing normalization-only, clipping-only, and combined approaches. Moving forward, we investigate existing variance-reduced algorithms, establishing that, in such a setting, normalization alone is sufficient for convergence. Finally, we present an accelerated variant that under second-order smoothness improves convergence. Our results provide theoretical insights and practical guidance for using normalization and clipping in nonconvex optimization with heavy-tailed noise.