Common TF-IDF variants arise as key components in the test statistic of a penalized likelihood-ratio test for word burstiness
This work provides a statistical interpretation of TF-IDF for researchers in natural language processing and information retrieval, but it is incremental as it offers insights rather than a new method.
The authors tackled the problem of understanding the statistical basis of TF-IDF by showing that TF-IDF-like scores emerge from a penalized likelihood-ratio test for word burstiness, with the alternative hypothesis using beta-binomial distributions and a gamma penalty. They found that this derived term-weighting scheme performs comparably to TF-IDF on document classification tasks.
TF-IDF is a classical formula that is widely used for identifying important terms within documents. We show that TF-IDF-like scores arise naturally from the test statistic of a penalized likelihood-ratio test setup capturing word burstiness (also known as word over-dispersion). In our framework, the alternative hypothesis captures word burstiness by modeling a collection of documents according to a family of beta-binomial distributions with a gamma penalty term on the precision parameter. In contrast, the null hypothesis assumes that words are binomially distributed in collection documents, a modeling approach that fails to account for word burstiness. We find that a term-weighting scheme given rise to by this test statistic performs comparably to TF-IDF on document classification tasks. This paper provides insights into TF-IDF from a statistical perspective and underscores the potential of hypothesis testing frameworks for advancing term-weighting scheme development.