Incremental Training of a Recurrent Neural Network Exploiting a Multi-Scale Dynamic Memory
This work addresses a known bottleneck in RNNs for tasks like speech recognition and handwriting analysis, but it is incremental as it builds on existing modular and multi-scale approaches.
The paper tackles the problem of capturing long-term dependencies in recurrent neural networks by proposing a modular architecture with incremental training that separates hidden states into modules operating at different frequencies. Experimental results on speech recognition and handwritten character datasets show improved ability to capture such dependencies.
The effectiveness of recurrent neural networks can be largely influenced by their ability to store into their dynamical memory information extracted from input sequences at different frequencies and timescales. Such a feature can be introduced into a neural architecture by an appropriate modularization of the dynamic memory. In this paper we propose a novel incrementally trained recurrent architecture targeting explicitly multi-scale learning. First, we show how to extend the architecture of a simple RNN by separating its hidden state into different modules, each subsampling the network hidden activations at different frequencies. Then, we discuss a training algorithm where new modules are iteratively added to the model to learn progressively longer dependencies. Each new module works at a slower frequency than the previous ones and it is initialized to encode the subsampled sequence of hidden activations. Experimental results on synthetic and real-world datasets on speech recognition and handwritten characters show that the modular architecture and the incremental training algorithm improve the ability of recurrent neural networks to capture long-term dependencies.