A Learn-to-Optimize Approach for Coordinate-Wise Step Sizes for Quasi-Newton Methods
This work addresses the problem of improving optimization efficiency for researchers and practitioners in machine learning, though it is incremental as it builds on existing quasi-Newton methods with a novel learning component.
The paper tackled the challenge of tuning coordinate-wise step sizes for quasi-Newton methods, which are under-explored and difficult in second-order contexts, and introduced a learn-to-optimize approach using LSTM networks that achieved up to 4x faster convergence in experiments.
Tuning step sizes is crucial for the stability and efficiency of optimization algorithms. While adaptive coordinate-wise step sizes have been shown to outperform scalar step size in first-order methods, their use in second-order methods is still under-explored and more challenging. Current approaches, including hypergradient descent and cutting plane methods, offer limited improvements or encounter difficulties in second-order contexts. To address these limitations, we first conduct a theoretical analysis within the Broyden-Fletcher-Goldfarb-Shanno (BFGS) framework, a prominent quasi-Newton method, and derive sufficient conditions for coordinate-wise step sizes that ensure convergence and stability. Building on this theoretical foundation, we introduce a novel learn-to-optimize (L2O) method that employs LSTM-based networks to learn optimal step sizes by leveraging insights from past optimization trajectories, while inherently respecting the derived theoretical guarantees. Extensive experiments demonstrate that our approach achieves substantial improvements over scalar step size methods and hypergradient descent-based method, offering up to 4$\times$ faster convergence across diverse optimization tasks.