Assessing the Memory Ability of Recurrent Neural Networks
This work addresses a theoretical and empirical gap in understanding RNN memory, which is incremental for improving the effectiveness and explainability of RNNs in sequence processing tasks.
The paper tackles the unclear memory abilities of different recurrent neural network (RNN) units by proposing a Semantic Euclidean Space to represent sequence semantics and defining evaluation indicators to measure and analyze these abilities, providing guidance for selecting sequence lengths during training.
It is known that Recurrent Neural Networks (RNNs) can remember, in their hidden layers, part of the semantic information expressed by a sequence (e.g., a sentence) that is being processed. Different types of recurrent units have been designed to enable RNNs to remember information over longer time spans. However, the memory abilities of different recurrent units are still theoretically and empirically unclear, thus limiting the development of more effective and explainable RNNs. To tackle the problem, in this paper, we identify and analyze the internal and external factors that affect the memory ability of RNNs, and propose a Semantic Euclidean Space to represent the semantics expressed by a sequence. Based on the Semantic Euclidean Space, a series of evaluation indicators are defined to measure the memory abilities of different recurrent units and analyze their limitations. These evaluation indicators also provide a useful guidance to select suitable sequence lengths for different RNNs during training.