Short-term Memory of Deep RNN
This work addresses memory limitations in RNNs for temporal data processing, offering insights for designing more efficient architectures, though it appears incremental in nature.
The paper investigates how layering in deep recurrent neural networks affects short-term memory, finding that higher layers inherently support longer memory spans even before training and that layered recurrent organization improves memory in reservoir computing models.
The extension of deep learning towards temporal data processing is gaining an increasing research interest. In this paper we investigate the properties of state dynamics developed in successive levels of deep recurrent neural networks (RNNs) in terms of short-term memory abilities. Our results reveal interesting insights that shed light on the nature of layering as a factor of RNN design. Noticeably, higher layers in a hierarchically organized RNN architecture results to be inherently biased towards longer memory spans even prior to training of the recurrent connections. Moreover, in the context of Reservoir Computing framework, our analysis also points out the benefit of a layered recurrent organization as an efficient approach to improve the memory skills of reservoir models.