Long-Short Range Context Neural Networks for Language Modeling
This work addresses the problem of capturing both syntactic and semantic dependencies in language modeling for natural language processing applications, representing an incremental improvement over existing methods.
The authors tackled language modeling by proposing a multi-span architecture that separately models short and long context information, dynamically merging them, resulting in a significant reduction in perplexity on the Penn Treebank and Large Text Compression Benchmark corpora compared to state-of-the-art techniques.
The goal of language modeling techniques is to capture the statistical and structural properties of natural languages from training corpora. This task typically involves the learning of short range dependencies, which generally model the syntactic properties of a language and/or long range dependencies, which are semantic in nature. We propose in this paper a new multi-span architecture, which separately models the short and long context information while it dynamically merges them to perform the language modeling task. This is done through a novel recurrent Long-Short Range Context (LSRC) network, which explicitly models the local (short) and global (long) context using two separate hidden states that evolve in time. This new architecture is an adaptation of the Long-Short Term Memory network (LSTM) to take into account the linguistic properties. Extensive experiments conducted on the Penn Treebank (PTB) and the Large Text Compression Benchmark (LTCB) corpus showed a significant reduction of the perplexity when compared to state-of-the-art language modeling techniques.