Encoding-based Memory Modules for Recurrent Neural Networks
This work addresses the memorization bottleneck in recurrent models for sequential tasks, offering an incremental improvement with modular memory and specialized training.
The paper tackled the problem of memorizing long sequences in recurrent neural networks by proposing a Linear Memory Network with an encoding-based memory component and a specialized training algorithm, showing that this approach improves performance on tasks requiring long-sequence memorization.
Learning to solve sequential tasks with recurrent models requires the ability to memorize long sequences and to extract task-relevant features from them. In this paper, we study the memorization subtask from the point of view of the design and training of recurrent neural networks. We propose a new model, the Linear Memory Network, which features an encoding-based memorization component built with a linear autoencoder for sequences. We extend the memorization component with a modular memory that encodes the hidden state sequence at different sampling frequencies. Additionally, we provide a specialized training algorithm that initializes the memory to efficiently encode the hidden activations of the network. The experimental results on synthetic and real-world datasets show that specializing the training algorithm to train the memorization component always improves the final performance whenever the memorization of long sequences is necessary to solve the problem.