Momentum Further Constrains Sharpness at the Edge of Stochastic Stability
For deep learning practitioners, this reveals that momentum and batch size jointly determine the implicit bias of optimization, with implications for hyperparameter tuning.
This work shows that SGD with momentum exhibits an Edge of Stochastic Stability regime with batch-size-dependent behavior, where sharpness stabilizes at two distinct plateaus: a lower plateau at small batch sizes favoring flatter regions, and a higher plateau at large batch sizes favoring sharper regions.
Recent work suggests that (stochastic) gradient descent self-organizes near an instability boundary, shaping both optimization and the solutions found. Momentum and mini-batch gradients are widely used in practical deep learning optimization, but it remains unclear whether they operate in a comparable regime of instability. We demonstrate that SGD with momentum exhibits an Edge of Stochastic Stability (EoSS)-like regime with batch-size-dependent behavior that cannot be explained by a single momentum-adjusted stability threshold. Batch Sharpness (the expected directional mini-batch curvature) stabilizes in two distinct regimes: at small batch sizes it converges to a lower plateau $2(1-β)/η$, reflecting amplification of stochastic fluctuations by momentum and favoring flatter regions than vanilla SGD; at large batch sizes it converges to a higher plateau $2(1+β)/η$, where momentum recovers its classical stabilizing effect and favors sharper regions consistent with full-batch dynamics. We further show that this aligns with linear stability thresholds and discuss the implications for hyperparameter tuning and coupling.