Exploring Differences Between Tabular Enterprise Data and Public Benchmarks
For practitioners and researchers working with enterprise tabular data, this work reveals the inadequacy of current benchmarks and the need for more representative evaluation.
This paper analyzes enterprise tabular data and finds it differs significantly from public benchmarks, showing that models performing well on benchmarks may perform poorly on enterprise data, highlighting the need for enterprise-specific benchmarks.
Tabular data dominate the landscape of data science, increasingly attracting innovative machine learning models and tailored benchmarks. Yet, little is known for enterprise data, where tables constitute the backbone of business operations. To broaden the benchmarking landscape for business applications, this work aims to actualize the characteristics of enterprise data by providing an analysis of data statistics and performance measurements of tabular models such as TabPFN, TabICL and ConTextTab. Through our analysis, we find enterprise data markedly differ from tabular benchmarks and we demonstrate that a tabular model that performs well on typical tabular benchmarks may perform poorly on real world enterprise data -- and vice versa. This lack of generalization underlines the need for additional benchmarks with enterprise-grade characteristics.