Context-Constrained Transfer Learning for Tabular Foundation Models via Data Distillation
Solves negative transfer in tabular foundation models for practitioners with limited target data and strict context windows.
TL-ANDI enables transfer learning for tabular foundation models under context-size constraints and distribution shifts, achieving up to 15% relative improvement over baselines on benchmark datasets.
Tabular Foundation Models (TFMs) have demonstrated strong empirical performance as black-box inference engines through in-context learning. However, their use in transfer learning is limited by two obstacles: strict context-size constraints and sensitivity to distribution shifts between source and target tasks. Directly pooling heterogeneous source data can therefore lead to negative transfer. To address these challenges, we propose Context-Constrained Transfer Learning via ANchoring and DIstillation (TL-ANDI), a posterior-aware distillation framework for TFMs. TL-ANDI constructs a compact source context by solving a budget-constrained optimal transport problem whose cost jointly measures target covariate coverage and posterior compatibility. The selected anchor samples are then equipped with locally distilled labels and combined with a residual calibration step using target data.