HyperAdam: A Learnable Task-Adaptive Adam for Network Training
This work addresses the need for more effective and generalizable optimizers in deep learning, offering an incremental improvement over existing methods by integrating learning-to-optimize techniques with traditional Adam.
The authors tackled the problem of improving deep neural network training by proposing HyperAdam, a learnable task-adaptive optimizer that combines multiple Adam updates with varying decay rates, achieving state-of-the-art performance on various networks like multilayer perceptrons, CNNs, and LSTMs.
Deep neural networks are traditionally trained using human-designed stochastic optimization algorithms, such as SGD and Adam. Recently, the approach of learning to optimize network parameters has emerged as a promising research topic. However, these learned black-box optimizers sometimes do not fully utilize the experience in human-designed optimizers, therefore have limitation in generalization ability. In this paper, a new optimizer, dubbed as \textit{HyperAdam}, is proposed that combines the idea of "learning to optimize" and traditional Adam optimizer. Given a network for training, its parameter update in each iteration generated by HyperAdam is an adaptive combination of multiple updates generated by Adam with varying decay rates. The combination weights and decay rates in HyperAdam are adaptively learned depending on the task. HyperAdam is modeled as a recurrent neural network with AdamCell, WeightCell and StateCell. It is justified to be state-of-the-art for various network training, such as multilayer perceptron, CNN and LSTM.