Outperforming Good-Turing: Preliminary Report
This work addresses a fundamental problem in machine learning and statistics for researchers and practitioners dealing with limited data, though it is presented as a preliminary report, indicating it may be incremental or exploratory.
The authors tackled the problem of estimating large alphabet probability distributions from limited samples by challenging the assumption that each symbol's sample frequency uniquely determines its estimated probability, and they reported a dramatic enhancement in estimation accuracy, typically up to 50%, compared to existing methods.
Estimating a large alphabet probability distribution from a limited number of samples is a fundamental problem in machine learning and statistics. A variety of estimation schemes have been proposed over the years, mostly inspired by the early work of Laplace and the seminal contribution of Good and Turing. One of the basic assumptions shared by most commonly-used estimators is the unique correspondence between the symbol's sample frequency and its estimated probability. In this work we tackle this paradigmatic assumption; we claim that symbols with "similar" frequencies shall be assigned the same estimated probability value. This way we regulate the number of parameters and improve generalization. In this preliminary report we show that by applying an ensemble of such regulated estimators, we introduce a dramatic enhancement in the estimation accuracy (typically up to 50%), compared to currently known methods. An implementation of our suggested method is publicly available at the first author's web-page.