Evaluating Latent Knowledge of Public Tabular Datasets in Large Language Models
This work addresses the problem of dataset contamination in LLM evaluations for researchers, highlighting an incremental but critical confound in assessing tabular reasoning.
The study investigated whether large language models (LLMs) exhibit prior knowledge of widely used tabular datasets like Adult Income and Titanic, revealing that contamination effects occur only for datasets with strong semantic cues, leading to near-random performance when cues are removed.
Large Language Models (LLMs) are increasingly evaluated on their ability to reason over structured data, yet such assessments often overlook a crucial confound: dataset contamination. In this work, we investigate whether LLMs exhibit prior knowledge of widely used tabular benchmarks such as Adult Income, Titanic, and others. Through a series of controlled probing experiments, we reveal that contamination effects emerge exclusively for datasets containing strong semantic cues-for instance, meaningful column names or interpretable value categories. In contrast, when such cues are removed or randomized, performance sharply declines to near-random levels. These findings suggest that LLMs' apparent competence on tabular reasoning tasks may, in part, reflect memorization of publicly available datasets rather than genuine generalization. We discuss implications for evaluation protocols and propose strategies to disentangle semantic leakage from authentic reasoning ability in future LLM assessments.