The hypergeometric test performs comparably to TF-IDF on standard text analysis tasks
This work addresses the need for theoretical justification of TF-IDF for researchers and practitioners in text analysis, though it is incremental as it builds on existing empirical comparisons.
The study tackled the problem of understanding the effectiveness of TF-IDF in text analysis by empirically comparing it to the hypergeometric test on document retrieval, summarization, and classification tasks, finding that they perform comparably.
Term frequency-inverse document frequency, or TF-IDF for short, and its many variants form a class of term weighting functions the members of which are widely used in text analysis applications. While TF-IDF was originally proposed as a heuristic, theoretical justifications grounded in information theory, probability, and the divergence from randomness paradigm have been advanced. In this work, we present an empirical study showing that TF-IDF corresponds very nearly with the hypergeometric test of statistical significance on selected real-data document retrieval, summarization, and classification tasks. These findings suggest that a fundamental mathematical connection between TF-IDF and the negative logarithm of the hypergeometric test P-value (i.e., a hypergeometric distribution tail probability) remains to be elucidated. We advance the empirical analyses herein as a first step toward explaining the long-standing effectiveness of TF-IDF from a statistical significance testing lens. It is our aspiration that these results will open the door to the systematic evaluation of significance testing derived term weighting functions in text analysis applications.