Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?
For the ML theory community, this open problem highlights a critical gap between practice and theory for a widely used optimizer.
The paper identifies a gap in the convergence theory of AdamW under heavy-tailed noise, which is common in LLM training, and formulates it as an open problem. It provides a positive weighted-metric benchmark and a lower-bound mechanism showing how denominator memory can hide large gradients.
AdamW is the de facto optimizer for training large language models (LLMs), yet the theory behind it still lives mostly in finite-variance regimes. This is increasingly unsatisfying, as empirical evidence indicates that stochastic gradient noise in LLM pretraining is typically heavy-tailed. Recent work shows that sign-based optimizers such as Lion and Muon achieve sharp heavy-tailed rates, and that AdaGrad can also converge under heavy-tailed noise. However, no rigorous convergence theory for AdamW has yet been established in this regime. Can AdamW converge under the same heavy-tailed assumptions, or does its second-moment accumulator create a genuine obstruction? We formulate this as an open problem, prove a positive weighted-metric benchmark, and give a corridor lower-bound mechanism showing how denominator memory can hide large gradients.