A memory enhanced LSTM for modeling complex temporal dependencies
This work addresses the challenge of handling long sequences in tasks like image and language processing, but it is incremental as it builds on existing LSTM architectures.
The paper tackles the problem of modeling complex temporal dependencies by introducing Gamma-LSTM, an enhanced LSTM unit with hierarchical memory, and demonstrates better performance than regular and stacked LSTMs in tasks like MNIST digit classification and natural language inference, emphasizing improved generalization over long sequences.
In this paper, we present Gamma-LSTM, an enhanced long short term memory (LSTM) unit, to enable learning of hierarchical representations through multiple stages of temporal abstractions. Gamma memory, a hierarchical memory unit, forms the central memory of Gamma-LSTM with gates to regulate the information flow into various levels of hierarchy, thus providing the unit with a control to pick the appropriate level of hierarchy to process the input at a given instant of time. We demonstrate better performance of Gamma-LSTM model regular and stacked LSTMs in two settings (pixel-by-pixel MNIST digit classification and natural language inference) placing emphasis on the ability to generalize over long sequences.