Benchmarks Are Not That Out of Distribution: Word Overlap Predicts Performance
This work addresses the issue of benchmark reliability for researchers and practitioners in NLP, revealing that many standard benchmarks are not truly out-of-distribution, which is incremental as it builds on existing data analysis methods.
The paper tackles the problem of understanding what drives benchmark performance in language models by investigating the role of word overlap between pre-training and evaluation data, finding a robust inverse relationship between word-level unigram cross-entropy and performance across 10 benchmarks, with results showing that larger subsets with similar statistics improve scores.
Understanding what constitutes high-quality pre-training data remains a central question in language model training. In this work, we investigate whether benchmark performance is primarily driven by the degree of statistical pattern overlap between pre-training corpora and evaluation datasets. We measure this overlap using word-level unigram cross-entropy and word frequency statistics, and perform controlled experiments across $10$ zero-shot benchmarks, $4$ pre-training datasets spanning $8.5\mathrm{B}$ to $60\mathrm{B}$ tokens, and model sizes ranging from $400\mathrm{M}$ to $3\mathrm{B}$ parameters. Our results demonstrate a robust inverse relationship between word-level unigram cross-entropy and benchmark performance, suggesting that widely used benchmarks are strongly influenced by word overlap between training and evaluation data. Thus, larger pre-training subsets with similar word-level unigram cross-entropy yield improved downstream results, indicating that word frequency statistics play an additional role in shaping benchmark scores. Taken together, these results suggest that many standard benchmarks are only weakly out-of-distribution relative to pre-training corpora, so that simple word-overlap statistics predict benchmark performance.