MADA: Meta-Adaptive Optimizers through hyper-gradient Descent
This addresses the need for more robust and adaptable optimizers in deep learning, though it is incremental as it builds on existing adaptive optimizer concepts.
The paper tackles the problem of adaptive optimizers in deep learning not performing uniformly well across all tasks by introducing MADA, a meta-adaptive optimizer framework that dynamically learns the most suitable optimizer during training, consistently outperforming Adam and others on vision and language tasks with improved validation performance in GPT-2 training.
Following the introduction of Adam, several novel adaptive optimizers for deep learning have been proposed. These optimizers typically excel in some tasks but may not outperform Adam uniformly across all tasks. In this work, we introduce Meta-Adaptive Optimizers (MADA), a unified optimizer framework that can generalize several known optimizers and dynamically learn the most suitable one during training. The key idea in MADA is to parameterize the space of optimizers and dynamically search through it using hyper-gradient descent during training. We empirically compare MADA to other popular optimizers on vision and language tasks, and find that MADA consistently outperforms Adam and other popular optimizers, and is robust against sub-optimally tuned hyper-parameters. MADA achieves a greater validation performance improvement over Adam compared to other popular optimizers during GPT-2 training and fine-tuning. We also propose AVGrad, a modification of AMSGrad that replaces the maximum operator with averaging, which is more suitable for hyper-gradient optimization. Finally, we provide a convergence analysis to show that parameterized interpolations of optimizers can improve their error bounds (up to constants), hinting at an advantage for meta-optimizers.