Overcoming the vanishing gradient problem in plain recurrent networks
This addresses a fundamental issue in sequence learning for researchers and practitioners, offering a simpler alternative to gated networks like LSTMs and GRUs.
The paper tackles the vanishing gradient problem in plain recurrent networks by proposing the Recurrent Identity Network (RIN), which achieves competitive performance and faster convergence, with small RIN models showing 12%–67% higher accuracy on Sequential and Permuted MNIST datasets and state-of-the-art results on the bAbI dataset.
Plain recurrent networks greatly suffer from the vanishing gradient problem while Gated Neural Networks (GNNs) such as Long-short Term Memory (LSTM) and Gated Recurrent Unit (GRU) deliver promising results in many sequence learning tasks through sophisticated network designs. This paper shows how we can address this problem in a plain recurrent network by analyzing the gating mechanisms in GNNs. We propose a novel network called the Recurrent Identity Network (RIN) which allows a plain recurrent network to overcome the vanishing gradient problem while training very deep models without the use of gates. We compare this model with IRNNs and LSTMs on multiple sequence modeling benchmarks. The RINs demonstrate competitive performance and converge faster in all tasks. Notably, small RIN models produce 12%--67% higher accuracy on the Sequential and Permuted MNIST datasets and reach state-of-the-art performance on the bAbI question answering dataset.