A Fixed-Size Encoding Method for Variable-Length Sequences with its Application to Neural Network Language Models
This work addresses the challenge of handling variable-length sequences in neural network language models for natural language processing, offering a novel encoding approach that improves performance over existing methods.
The paper tackles the problem of encoding variable-length sequences into fixed-size representations by proposing the Fixed-Size Ordinally-Forgetting Encoding (FOFE) method, which uses a simple ordinally-forgetting mechanism to model word order. Experimental results show that FOFE-based feedforward neural network language models significantly outperform standard fixed-input models and popular recurrent neural network language models without using recurrent feedbacks.
In this paper, we propose the new fixed-size ordinally-forgetting encoding (FOFE) method, which can almost uniquely encode any variable-length sequence of words into a fixed-size representation. FOFE can model the word order in a sequence using a simple ordinally-forgetting mechanism according to the positions of words. In this work, we have applied FOFE to feedforward neural network language models (FNN-LMs). Experimental results have shown that without using any recurrent feedbacks, FOFE based FNN-LMs can significantly outperform not only the standard fixed-input FNN-LMs but also the popular RNN-LMs.