Table Foundation Models: on knowledge pre-training for tabular learning
This work addresses the problem of inefficient and non-reusable table foundation models for data scientists, offering a more convenient and effective approach, though it appears incremental by building on existing pre-trained neural networks.
The paper tackles the challenge of data semantics in tabular learning by introducing TARTE, a foundation model that transforms tables to knowledge-enhanced vector representations using strings, pre-trained on large relational data, resulting in state-of-the-art prediction performance and improved prediction/computation trade-off.
Table foundation models bring high hopes to data science: pre-trained on tabular data to embark knowledge or priors, they should facilitate downstream tasks on tables. One specific challenge is that of data semantics: numerical entries take their meaning from context, e.g., column name. Pre-trained neural networks that jointly model column names and table entries have recently boosted prediction accuracy. While these models outline the promises of world knowledge to interpret table values, they lack the convenience of popular foundation models in text or vision. Indeed, they must be fine-tuned to bring benefits, come with sizeable computation costs, and cannot easily be reused or combined with other architectures. Here we introduce TARTE, a foundation model that transforms tables to knowledge-enhanced vector representations using the string to capture semantics. Pre-trained on large relational data, TARTE yields representations that facilitate subsequent learning with little additional cost. These representations can be fine-tuned or combined with other learners, giving models that push the state-of-the-art prediction performance and improve the prediction/computation performance trade-off. Specialized to a task or a domain, TARTE gives domain-specific representations that facilitate further learning. Our study demonstrates an effective approach to knowledge pre-training for tabular learning.