Learning Gradient Descent: Better Generalization and Longer Horizons
This addresses the problem of automating hyperparameter tuning for deep learning practitioners, though it appears incremental as it builds on existing learning-to-learn approaches.
The paper tackles the labor-intensive process of tuning deep neural network training by proposing a new learning-to-learn optimizer that outperforms both generic hand-crafted algorithms and state-of-the-art learning-to-learn methods from DeepMind across tasks like MLPs, CNNs, and LSTMs.
Training deep neural networks is a highly nontrivial task, involving carefully selecting appropriate training algorithms, scheduling step sizes and tuning other hyperparameters. Trying different combinations can be quite labor-intensive and time consuming. Recently, researchers have tried to use deep learning algorithms to exploit the landscape of the loss function of the training problem of interest, and learn how to optimize over it in an automatic way. In this paper, we propose a new learning-to-learn model and some useful and practical tricks. Our optimizer outperforms generic, hand-crafted optimization algorithms and state-of-the-art learning-to-learn optimizers by DeepMind in many tasks. We demonstrate the effectiveness of our algorithms on a number of tasks, including deep MLPs, CNNs, and simple LSTMs.