Learning Provably Improves the Convergence of Gradient Descent
This work addresses a theoretical gap for researchers in optimization and machine learning, providing provable convergence for L2O, though it is incremental as it focuses on a specific problem class.
The paper tackles the lack of theoretical convergence guarantees for Learn to Optimize (L2O) methods by proving training convergence for L2O models that learn Gradient Descent hyperparameters in quadratic programming, using Neural Tangent Kernel theory, and demonstrates over 50% better optimality than standard Gradient Descent.
Learn to Optimize (L2O) trains deep neural network-based solvers for optimization, achieving success in accelerating convex problems and improving non-convex solutions. However, L2O lacks rigorous theoretical backing for its own training convergence, as existing analyses often use unrealistic assumptions -- a gap this work highlights empirically. We bridge this gap by proving the training convergence of L2O models that learn Gradient Descent (GD) hyperparameters for quadratic programming, leveraging the Neural Tangent Kernel (NTK) theory. We propose a deterministic initialization strategy to support our theoretical results and promote stable training over extended optimization horizons by mitigating gradient explosion. Our L2O framework demonstrates over 50% better optimality than GD and superior robustness over state-of-the-art L2O methods on synthetic datasets. The code of our method can be found from https://github.com/NetX-lab/MathL2OProof-Official.