adaQN: An Adaptive Quasi-Newton Algorithm for Training RNNs
This addresses the problem of efficient RNN training for practitioners, but it is incremental as it builds on existing quasi-Newton methods with specific adaptations.
The paper tackles the computationally difficult training of RNNs due to vanishing/exploding gradients by proposing adaQN, a stochastic quasi-Newton algorithm that retains low per-iteration cost while allowing non-diagonal scaling, and shows it is competitive with popular RNN training algorithms on language modeling tasks.
Recurrent Neural Networks (RNNs) are powerful models that achieve exceptional performance on several pattern recognition problems. However, the training of RNNs is a computationally difficult task owing to the well-known "vanishing/exploding" gradient problem. Algorithms proposed for training RNNs either exploit no (or limited) curvature information and have cheap per-iteration complexity, or attempt to gain significant curvature information at the cost of increased per-iteration cost. The former set includes diagonally-scaled first-order methods such as ADAGRAD and ADAM, while the latter consists of second-order algorithms like Hessian-Free Newton and K-FAC. In this paper, we present adaQN, a stochastic quasi-Newton algorithm for training RNNs. Our approach retains a low per-iteration cost while allowing for non-diagonal scaling through a stochastic L-BFGS updating scheme. The method uses a novel L-BFGS scaling initialization scheme and is judicious in storing and retaining L-BFGS curvature pairs. We present numerical experiments on two language modeling tasks and show that adaQN is competitive with popular RNN training algorithms.