Scaling Experiments in Self-Supervised Cross-Table Representation Learning
This addresses scaling challenges in tabular data representation learning for machine learning practitioners, though it appears incremental as it builds on existing self-supervised and Transformer approaches.
The paper tackled the problem of scaling deep tabular representation learning models by introducing a Transformer-based architecture for cross-table learning, trained via self-supervised masked cell recovery. The result showed models with 10^4 to 10^7 parameters trained on 135M tokens from 76 datasets, evaluated through linear probing on benchmarks.
To analyze the scaling potential of deep tabular representation learning models, we introduce a novel Transformer-based architecture specifically tailored to tabular data and cross-table representation learning by utilizing table-specific tokenizers and a shared Transformer backbone. Our training approach encompasses both single-table and cross-table models, trained via missing value imputation through a self-supervised masked cell recovery objective. To understand the scaling behavior of our method, we train models of varying sizes, ranging from approximately $10^4$ to $10^7$ parameters. These models are trained on a carefully curated pretraining dataset, consisting of 135M training tokens sourced from 76 diverse datasets. We assess the scaling of our architecture in both single-table and cross-table pretraining setups by evaluating the pretrained models using linear probing on a curated set of benchmark datasets and comparing the results with conventional baselines.