Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models
For researchers studying tabular foundation models, this work provides initial understanding of their inference mechanisms and demonstrates potential for parameter efficiency.
This paper presents the first mechanistic study of layerwise inference dynamics in tabular foundation models, revealing depthwise redundancy. Based on these insights, they design a looped single-layer model that uses only 20% of the original parameters while achieving comparable performance.
Transformer-based tabular foundation models (TFMs) dominate small to medium tabular predictive benchmark tasks, yet their inference mechanisms remain largely unexplored. We present the first large-scale mechanistic study of layerwise dynamics in 6 state-of-the-art tabular in-context learning models. We explore how predictions emerge across depth, identify distinct stages of inference and reveal latent-space dynamics that differ from those of language models. Our findings indicate substantial depthwise redundancy across multiple models, suggesting iterative refinement with overlapping computations during inference stages. Guided by these insights, we design a proof-of-concept, looped single-layer model that uses only 20% of the original model's parameters while achieving comparable performance. The code is available at https://github.com/amirbalef/is_one_layer_enough.