A Generalizable Approach to Learning Optimizers
This addresses a core issue in machine learning by improving optimizer generalization across diverse neural network tasks, though it appears incremental as it builds on existing learning-to-optimize methods.
The paper tackles the problem of learning optimizers that generalize poorly to real-world tasks by proposing a system that learns to update optimizer hyperparameters, achieving 2x speedups on ImageNet and 2.5x speedups on a language modeling task with significantly more compute than training tasks.
A core issue with learning to optimize neural networks has been the lack of generalization to real world problems. To address this, we describe a system designed from a generalization-first perspective, learning to update optimizer hyperparameters instead of model parameters directly using novel features, actions, and a reward function. This system outperforms Adam at all neural network tasks including on modalities not seen during training. We achieve 2x speedups on ImageNet, and a 2.5x speedup on a language modeling task using over 5 orders of magnitude more compute than the training tasks.