Where Computation Lives Inside TabPFN: Causal Localisation of Attention Head Function
For researchers studying mechanistic interpretability of tabular models, this work provides initial causal insights into TabPFN's attention heads, though it is incremental as it applies existing methods (activation patching, ablation) to a new model.
This paper presents the first causal mechanistic analysis of a tabular foundation model, TabPFN 2.5, revealing that one attention head dominates causal necessity by 2-5× at its peak layer, with its dominant layer shifting based on task complexity. Contrastive activation steering fails to transfer across samples due to TabPFN's in-context learning mechanism.
We present the first causal mechanistic analysis of a tabular foundation model, investigating how TabPFN 2.5's feature wise attention heads distribute computation across layers. Using activation patching, ablation, and attention entropy across two synthetic regression datasets, we find clear temporal specialisation: one head's causal necessity dominates that of the others by 2 to 5 times at peak layer, with its dominant layer shifting across tasks of different complexity, while the remaining heads exhibit symmetric late layer profiles. Attention entropy and patching provide convergent evidence for the computationally active layers of the dominant head. We additionally investigate inference time steerability via contrastive activation steering, which fails to transfer across samples. We attribute this result to TabPFN's in context learning mechanism, which encodes task structure through context dependent attention rather than the stable parametric directions that make steering tractable in language models.