End-to-End Compression for Tabular Foundation Models
This addresses efficiency bottlenecks for researchers and practitioners using tabular foundation models, enabling handling of larger datasets with reduced computational overhead, though it is incremental as it builds on existing transformer architectures.
The paper tackles the quadratic complexity and high memory consumption of transformer-based tabular foundation models by proposing TACO, an end-to-end compression model that compresses training data into a latent space, achieving up to 94x faster inference and 97% less memory usage on the TabArena benchmark while maintaining performance.
The long-standing dominance of gradient-boosted decision trees for tabular data has recently been challenged by in-context learning tabular foundation models. In-context learning methods fit and predict in one forward pass without parameter updates by leveraging the training data as context for predicting on query test points. While recent tabular foundation models achieve state-of-the-art performance, their transformer architecture based on the attention mechanism has quadratic complexity regarding dataset size, which in turn increases the overhead on training and inference time, and limits the capacity of the models to handle large-scale datasets. In this work, we propose TACO, an end-to-end tabular compression model that compresses the training dataset in a latent space. We test our method on the TabArena benchmark, where our proposed method is up to 94x faster in inference time, while consuming up to 97\% less memory compared to the state-of-the-art tabular transformer architecture, all while retaining performance without significant degradation. Lastly, our method not only scales better with increased dataset sizes, but it also achieves better performance compared to other baselines.