Feature Selective Likelihood Ratio Estimator for Low- and Zero-frequency N-grams
This addresses a specific bottleneck in NLP for tasks involving rare N-grams, but it is incremental as it builds on existing decomposition and feature selection techniques.
The paper tackles the problem of estimating likelihood ratios for low- and zero-frequency N-grams in NLP by proposing a method that combines decomposition with feature selection, resulting in effective and efficient estimation.
In natural language processing (NLP), the likelihood ratios (LRs) of N-grams are often estimated from the frequency information. However, a corpus contains only a fraction of the possible N-grams, and most of them occur infrequently. Hence, we desire an LR estimator for low- and zero-frequency N-grams. One way to achieve this is to decompose the N-grams into discrete values, such as letters and words, and take the product of the LRs for the values. However, because this method deals with a large number of discrete values, the running time and memory usage for estimation are problematic. Moreover, use of unnecessary discrete values causes deterioration of the estimation accuracy. Therefore, this paper proposes combining the aforementioned method with the feature selection method used in document classification, and shows that our estimator provides effective and efficient estimation results for low- and zero-frequency N-grams.