EquiTabPFN: A Target-Permutation Equivariant Prior Fitted Networks
This work addresses instability issues in tabular data models for machine learning practitioners, though it is incremental as it builds on prior models like TabPFN.
The paper tackled the problem of foundational models for tabular data lacking target equivariance, which causes instability in predictions due to an 'equivariance gap', and resulted in a model that matches or surpasses existing methods on datasets with more classes than seen during pre-training while reducing computational overhead.
Recent foundational models for tabular data, such as TabPFN, excel at adapting to new tasks via in-context learning, but remain constrained to a fixed, pre-defined number of target dimensions-often necessitating costly ensembling strategies. We trace this constraint to a deeper architectural shortcoming: these models lack target equivariance, so that permuting target dimension orderings alters their predictions. This deficiency gives rise to an irreducible "equivariance gap", an error term that introduces instability in predictions. We eliminate this gap by designing a fully target-equivariant architecture-ensuring permutation invariance via equivariant encoders, decoders, and a bi-attention mechanism. Empirical evaluation on standard classification benchmarks shows that, on datasets with more classes than those seen during pre-training, our model matches or surpasses existing methods while incurring lower computational overhead.