Safeguarding adaptive methods: global convergence of Barzilai-Borwein and other stepsize choices
This work addresses convergence guarantees for optimization algorithms in machine learning and applied mathematics, though it appears incremental as it refines existing results.
The paper tackles the problem of ensuring global convergence for adaptive optimization methods like Barzilai-Borwein stepsize in convex minimization, even when gradients are only locally Hölder continuous, by introducing a linesearch-free proximal gradient framework, with numerical evidence supporting the synergy between fast stepsize choices and adaptive methods.
Leveraging on recent advancements on adaptive methods for convex minimization problems, this paper provides a linesearch-free proximal gradient framework for globalizing the convergence of popular stepsize choices such as Barzilai-Borwein and one-dimensional Anderson acceleration. This framework can cope with problems in which the gradient of the differentiable function is merely locally Hölder continuous. Our analysis not only encompasses but also refines existing results upon which it builds. The theory is corroborated by numerical evidence that showcases the synergetic interplay between fast stepsize selections and adaptive methods.