A Generalized Language Model as the Combination of Skipped n-grams and Modified Kneser-Ney Smoothing
This work addresses the challenge of improving language modeling accuracy, particularly for sparse training data, which is incremental as it builds on existing smoothing techniques.
The paper tackles the problem of building more effective language models by introducing a generalized approach that combines skipped n-grams with modified Kneser-Ney smoothing, resulting in perplexity reductions of 3.1% to 12.7% on English text and up to 25.7% on sparse data.
We introduce a novel approach for building language models based on a systematic, recursive exploration of skip n-gram models which are interpolated using modified Kneser-Ney smoothing. Our approach generalizes language models as it contains the classical interpolation with lower order models as a special case. In this paper we motivate, formalize and present our approach. In an extensive empirical experiment over English text corpora we demonstrate that our generalized language models lead to a substantial reduction of perplexity between 3.1% and 12.7% in comparison to traditional language models using modified Kneser-Ney smoothing. Furthermore, we investigate the behaviour over three other languages and a domain specific corpus where we observed consistent improvements. Finally, we also show that the strength of our approach lies in its ability to cope in particular with sparse training data. Using a very small training data set of only 736 KB text we yield improvements of even 25.7% reduction of perplexity.