Feedforward Sequential Memory Neural Networks without Recurrent Feedback
This work addresses the challenge of capturing long-term dependencies in sequence modeling for language tasks, offering a novel alternative to recurrent architectures.
The authors tackled the problem of learning long-term dependencies in language modeling by introducing feedforward sequential memory networks (FSMN), which avoid recurrent feedback. Experimental results showed that FSMN-based language models significantly outperformed both feedforward and recurrent neural network models on several tasks.
We introduce a new structure for memory neural networks, called feedforward sequential memory networks (FSMN), which can learn long-term dependency without using recurrent feedback. The proposed FSMN is a standard feedforward neural networks equipped with learnable sequential memory blocks in the hidden layers. In this work, we have applied FSMN to several language modeling (LM) tasks. Experimental results have shown that the memory blocks in FSMN can learn effective representations of long history. Experiments have shown that FSMN based language models can significantly outperform not only feedforward neural network (FNN) based LMs but also the popular recurrent neural network (RNN) LMs.