Hierarchically Gated Recurrent Neural Network for Sequence Modeling
This work addresses the need for more efficient and effective sequence models, particularly for tasks requiring long-term dependency modeling, though it appears incremental as it builds on existing linear RNN approaches.
The paper tackles the problem of improving linear RNNs for sequence modeling by introducing a hierarchically gated mechanism with forget gates, resulting in enhanced performance on language modeling, image classification, and long-range benchmarks.
Transformers have surpassed RNNs in popularity due to their superior abilities in parallel training and long-term dependency modeling. Recently, there has been a renewed interest in using linear RNNs for efficient sequence modeling. These linear RNNs often employ gating mechanisms in the output of the linear recurrence layer while ignoring the significance of using forget gates within the recurrence. In this paper, we propose a gated linear RNN model dubbed Hierarchically Gated Recurrent Neural Network (HGRN), which includes forget gates that are lower bounded by a learnable value. The lower bound increases monotonically when moving up layers. This allows the upper layers to model long-term dependencies and the lower layers to model more local, short-term dependencies. Experiments on language modeling, image classification, and long-range arena benchmarks showcase the efficiency and effectiveness of our proposed model. The source code is available at https://github.com/OpenNLPLab/HGRN.