Active In-Context Learning for Tabular Foundation Models
Provides a practical active learning method for tabular data that overcomes cold-start limitations, benefiting practitioners with limited labeling budgets.
Active learning for tabular data suffers from unreliable uncertainty estimates in cold-start settings. The authors propose Tab-AICL, which uses TabPFN's in-context learning to iteratively select informative samples, achieving up to 100% normalized AULC improvement over gradient-boosting baselines on 20 benchmarks.
Active learning (AL) reduces labeling cost by querying informative samples, but in tabular settings its cold-start gains are often limited because uncertainty estimates are unreliable when models are trained on very few labels. Tabular foundation models such as TabPFN provide calibrated probabilistic predictions via in-context learning (ICL), i.e., without task-specific weight updates, enabling an AL regime in which the labeled context - rather than parameters - is iteratively optimized. We formalize Tabular Active In-Context Learning (Tab-AICL) and instantiate it with four acquisition rules: uncertainty (TabPFN-Margin), diversity (TabPFN-Coreset), an uncertainty-diversity hybrid (TabPFN-Hybrid), and a scalable two-stage method (TabPFN-Proxy-Hybrid) that shortlists candidates using a lightweight linear proxy before TabPFN-based selection. Across 20 classification benchmarks, Tab-AICL improves cold-start sample efficiency over retrained gradient-boosting baselines (CatBoost-Margin and XGBoost-Margin), measured by normalized AULC up to 100 labeled samples.