A Novel Framework for Recurrent Neural Networks with Enhancing Information Processing and Transmission between Units
This work addresses a specific bottleneck in RNNs for researchers and practitioners, but it appears incremental as it builds on existing memory concepts without introducing a new paradigm.
The paper tackles the problem of inefficient information processing in recurrent neural networks by proposing a novel framework inspired by human memory models, resulting in improved performance across text classification, image classification, and language modeling tasks on 6 datasets.
This paper proposes a novel framework for recurrent neural networks (RNNs) inspired by the human memory models in the field of cognitive neuroscience to enhance information processing and transmission between adjacent RNNs' units. The proposed framework for RNNs consists of three stages that is working memory, forget, and long-term store. The first stage includes taking input data into sensory memory and transferring it to working memory for preliminary treatment. And the second stage mainly focuses on proactively forgetting the secondary information rather than the primary in the working memory. And finally, we get the long-term store normally using some kind of RNN's unit. Our framework, which is generalized and simple, is evaluated on 6 datasets which fall into 3 different tasks, corresponding to text classification, image classification and language modelling. Experiments reveal that our framework can obviously improve the performance of traditional recurrent neural networks. And exploratory task shows the ability of our framework of correctly forgetting the secondary information.