Linear Memory Networks
This work addresses a foundational issue in recurrent neural network design for sequence modeling tasks, offering an incremental improvement with potential applications in domains like music processing.
The paper tackles the problem of disentangling memory and input-output functions in recurrent neural networks by introducing Linear Memory Networks, which separate these components into a feedforward network and a linear autoencoder, achieving competitive results on polyphonic music datasets.
Recurrent neural networks can learn complex transduction problems that require maintaining and actively exploiting a memory of their inputs. Such models traditionally consider memory and input-output functionalities indissolubly entangled. We introduce a novel recurrent architecture based on the conceptual separation between the functional input-output transformation and the memory mechanism, showing how they can be implemented through different neural components. By building on such conceptualization, we introduce the Linear Memory Network, a recurrent model comprising a feedforward neural network, realizing the non-linear functional transformation, and a linear autoencoder for sequences, implementing the memory component. The resulting architecture can be efficiently trained by building on closed-form solutions to linear optimization problems. Further, by exploiting equivalence results between feedforward and recurrent neural networks we devise a pretraining schema for the proposed architecture. Experiments on polyphonic music datasets show competitive results against gated recurrent networks and other state of the art models.