Meta-Learning Parameterized First-Order Optimizers using Differentiable Convex Optimization
This work addresses the challenge of automating optimizer selection for machine learning and control tasks, offering a novel meta-learning approach that could reduce reliance on trial-and-error.
The paper tackles the problem of selecting optimization methods and hyperparameters by proposing a meta-learning framework that uses differentiable convex optimization (DCO) to generalize first-order update rules, achieving one-step optimization for linear least squares problems with sufficient task exposure.
Conventional optimization methods in machine learning and controls rely heavily on first-order update rules. Selecting the right method and hyperparameters for a particular task often involves trial-and-error or practitioner intuition, motivating the field of meta-learning. We generalize a broad family of preexisting update rules by proposing a meta-learning framework in which the inner loop optimization step involves solving a differentiable convex optimization (DCO). We illustrate the theoretical appeal of this approach by showing that it enables one-step optimization of a family of linear least squares problems, given that the meta-learner has sufficient exposure to similar tasks. Various instantiations of the DCO update rule are compared to conventional optimizers on a range of illustrative experimental settings.