KnowsTFM: Knowledge-Informed Fine-Tuning of Small Tabular Foundation Models
For practitioners in niche domains with relational knowledge, this provides a method to adapt small tabular foundation models, though gains are incremental and limited to specialist settings.
KnowsTFM introduces knowledge-informed fine-tuning for small tabular foundation models, using structural attention priors from knowledge graphs and low-rank updates. It achieves meaningful gains in niche domains with scarce, high-dimensional data, but only marginal improvements on general-domain tasks.
Tabular foundation models have advanced deep learning for tabular data by delivering strong default performance across many small and medium tasks. Yet in niche domains, where data is scarce, high-dimensional, and shifted from the pretraining distribution, they may still fail to outperform carefully designed domain-specific methods. Many such domains also provide curated relational knowledge in the form of knowledge graphs and knowledge banks, but how to use this knowledge to improve and steer \textit{small} specialist tabular foundation models remains unclear. We address this problem through \textbf{Know}ledge-informed fine-tuning of \textbf{s}mall \textbf{T}abular \textbf{F}oundation \textbf{M}odels (\modelname). Specifically, we study nanoscale TabPFN- and TabICL-style variants, pretrained under controlled synthetic prior families and adapted using two complementary mechanisms: structural attention priors derived from knowledge graphs and parameter-efficient low-rank updates. We show that injecting domain-specific structural knowledge during fine-tuning yields meaningful gains over vanilla variants in specialist settings, whereas gains on general-domain tasks are marginal. We further observe that continual fine-tuning of frontier models can trigger collapse of pretrained knowledge and mechanisms.