The Illusion of Generalization: Re-examining Tabular Language Model Evaluation
This work challenges the validity of generalization claims in tabular language models, highlighting critical evaluation flaws that could mislead the ML community.
The authors re-evaluated Tabular Language Models (TLMs) using 165 datasets and found that claimed generalization is largely due to evaluation artifacts, such as dataset contamination and task-specific quirks, rather than learned tabular reasoning, with instruction-tuning recovering 92.2% of performance without tabular exposure.
Tabular Language Models (TLMs) have been claimed to achieve emergent generalization for tabular prediction. We conduct a systematic re-evaluation of Tabula-8B as a representative TLM, utilizing 165 datasets from the UniPredict benchmark. Our investigation reveals three findings. First, binary and categorical classification achieve near-zero median lift over majority-class baselines and strong aggregate performance is driven entirely by quartile classification tasks. Second, top-performing datasets exhibit pervasive contamination, including complete train-test overlap and task-level leakage that evades standard deduplication. Third, instruction-tuning without tabular exposure recovers 92.2% of standard classification performance and on quartile classification, format familiarity closes 71.3% of the gap with the residual attributable to contaminated datasets. These findings suggest claimed generalization likely reflects evaluation artifacts rather than learned tabular reasoning. We conclude with recommendations for strengthening TLM evaluation.