GOTabPFN: From Feature Ordering to Compact Tokenization for Tabular Foundation Models on High-Dimensional Data
This work addresses the problem of applying small tabular foundation models to high-dimensional data, a known bottleneck for such models, but the improvements are incremental.
GOTabPFN introduces a feature ordering method (GO-LR) and a compression unit (NSC) to enable small tabular foundation models to handle high-dimensional, low-sample-size (HDLSS) data without retraining, improving stability and accuracy under tight token budgets.
We investigate how to make small tabular foundation models effective for High-Dimensional, Low-Sample Size (HDLSS) tabular prediction without retraining large backbones. We introduce Graph-guided Ordering with Local Refinement (GO-LR), show its equivalence to weighted Minimum Linear Arrangement, and interpret the practical solver as a TSP-path-style surrogate. We propose GOTabPFN,which builds on GO-LR, and a Neuro-Inspired Subunit Compression (NSC) unit to pool locally adjacent ordered features into meta-features, yielding a compact representation that makes TabPFN-style prediction practical in HDLSS regimes. Across tabular benchmarks, GOTabPFN improves stability and accuracy under tight token budgets.