A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases
This work addresses the problem of making learned optimizers more stable and generalizable for machine learning practitioners, representing an incremental improvement over existing methods.
The paper tackled the instability and poor generalization of blackbox learned optimizers by analyzing their inductive biases and stability using dynamical systems, and introduced architectural and training modifications that improved stability and inductive bias. The resulting optimizer outperformed the state-of-the-art learned optimizer in optimization performance and meta-training speed, with better generalization to unseen tasks.
Learned optimizers -- neural networks that are trained to act as optimizers -- have the potential to dramatically accelerate training of machine learning models. However, even when meta-trained across thousands of tasks at huge computational expense, blackbox learned optimizers often struggle with stability and generalization when applied to tasks unlike those in their meta-training set. In this paper, we use tools from dynamical systems to investigate the inductive biases and stability properties of optimization algorithms, and apply the resulting insights to designing inductive biases for blackbox optimizers. Our investigation begins with a noisy quadratic model, where we characterize conditions in which optimization is stable, in terms of eigenvalues of the training dynamics. We then introduce simple modifications to a learned optimizer's architecture and meta-training procedure which lead to improved stability, and improve the optimizer's inductive bias. We apply the resulting learned optimizer to a variety of neural network training tasks, where it outperforms the current state of the art learned optimizer -- at matched optimizer computational overhead -- with regard to optimization performance and meta-training speed, and is capable of generalization to tasks far different from those it was meta-trained on.