On the Interplay Between Stepsize Tuning and Progressive Sharpening
This work addresses optimization challenges for deep learning practitioners by revealing limitations of classical stepsize tuners and highlighting better-performing alternatives, though it is incremental as it builds on prior empirical findings.
The paper investigates how stepsize tuners like Armijo linesearch and Polyak stepsizes affect the sharpness evolution in deep learning optimization, finding that Armijo linesearch performs poorly due to increasing sharpness, while Polyak stepsizes operate at or beyond the edge of stability and outperform other methods in deterministic settings.
Recent empirical work has revealed an intriguing property of deep learning models by which the sharpness (largest eigenvalue of the Hessian) increases throughout optimization until it stabilizes around a critical value at which the optimizer operates at the edge of stability, given a fixed stepsize (Cohen et al, 2022). We investigate empirically how the sharpness evolves when using stepsize-tuners, the Armijo linesearch and Polyak stepsizes, that adapt the stepsize along the iterations to local quantities such as, implicitly, the sharpness itself. We find that the surprisingly poor performance of a classical Armijo linesearch in the deterministic setting may be well explained by its tendency to ever-increase the sharpness of the objective. On the other hand, we observe that Polyak stepsizes operate generally at the edge of stability or even slightly beyond, outperforming its Armijo and constant stepsizes counterparts in the deterministic setting. We conclude with an analysis that suggests unlocking stepsize tuners requires an understanding of the joint dynamics of the step size and the sharpness.