Anytime Acceleration of Gradient Descent
It solves an open problem in optimization theory by providing stepsize-based acceleration with anytime guarantees, which is incremental but addresses a specific theoretical bottleneck.
This work tackles the problem of achieving anytime convergence guarantees for gradient descent in smooth convex optimization, proposing a predetermined stepsize schedule that yields a convergence rate of O(T^{-1.119}) for any stopping time T, and extends this to strongly convex cases with exponential convergence.
This work investigates stepsize-based acceleration of gradient descent with {\em anytime} convergence guarantees. For smooth (non-strongly) convex optimization, we propose a stepsize schedule that allows gradient descent to achieve convergence guarantees of $O(T^{-1.119})$ for any stopping time $T$, where the stepsize schedule is predetermined without prior knowledge of the stopping time. This result provides an affirmative answer to a COLT open problem \citep{kornowski2024open} regarding whether stepsize-based acceleration can yield anytime convergence rates of $o(T^{-1})$. We further extend our theory to yield anytime convergence guarantees of $\exp(-Ω(T/κ^{0.893}))$ for smooth and strongly convex optimization, with $κ$ being the condition number.