A short proof of near-linear convergence of adaptive gradient descent under fourth-order growth and convexity
It simplifies the analysis of an existing algorithm for convex optimization problems with quartic growth, offering a more adaptive version.
The paper provides a simpler Lyapunov-based proof of near-linear convergence for adaptive gradient descent on smooth convex functions with fourth-order growth, and introduces a more adaptive variant with better numerical performance.
Davis, Drusvyatskiy, and Jiang showed that gradient descent with an adaptive stepsize converges locally at a nearly-linear rate for smooth functions that grow at least quartically away from their minimizers. The argument is intricate, relying on monitoring the performance of the algorithm relative to a certain manifold of slow growth -- called the ravine. In this work, we provide a direct Lyapunov-based argument that bypasses these difficulties when the objective is in addition convex and a has a unique minimizer. As a byproduct of the argument, we obtain a more adaptive variant than the original algorithm with encouraging numerical performance.