iLTM: Integrated Large Tabular Model
This work addresses the gap between tree-based and neural methods for tabular data, offering a new framework for robust and scalable learning in domains like science and industry, though it appears incremental as it builds on existing techniques.
The paper tackles the problem of applying deep learning to tabular data, where gradient-boosted decision trees (GBDTs) are still dominant, by introducing iLTM, an integrated Large Tabular Model that unifies various components and is pretrained on over 1,800 datasets; it achieves consistently superior performance in classification and regression tasks, outperforming GBDTs and other deep models with less tuning.
Tabular data underpins decisions across science, industry, and public services. Despite rapid progress, advances in deep learning have not fully carried over to the tabular domain, where gradient-boosted decision trees (GBDTs) remain a default choice in practice. We present iLTM, an integrated Large Tabular Model that unifies tree-derived embeddings, dimensionality-agnostic representations, a meta-trained hypernetwork, multilayer perceptrons (MLPs), and retrieval within a single architecture. Pretrained on more than 1,800 heterogeneous classification datasets, iLTM achieves consistently superior performance across tabular classification and regression tasks, from small datasets to large and high-dimensional tasks. After light fine-tuning, the meta-trained hypernetwork transfers to regression targets, matching or surpassing strong baselines. Extensive experiments show that iLTM outperforms well-tuned GBDTs and leading deep tabular models while requiring less task-specific tuning. By bridging the gap between tree-based and neural methods, iLTM offers a new framework for tabular foundation models for robust, adaptable, and scalable tabular learning.