Learning-To-Measure: In-context Active Feature Acquisition
This addresses scalability limitations in AFA for machine learning practitioners by enabling cross-task learning without per-task retraining, though it is incremental as it builds on existing AFA methods.
The paper tackles the problem of active feature acquisition (AFA) by formalizing a meta-AFA approach to learn acquisition policies across various tasks, introducing Learning-to-Measure (L2M) with uncertainty quantification and a greedy agent, and shows it matches or surpasses task-specific baselines, especially under scarce labels and high missingness.
Active feature acquisition (AFA) is a sequential decision-making problem where the goal is to improve model performance for test instances by adaptively selecting which features to acquire. In practice, AFA methods often learn from retrospective data with systematic missingness in the features and limited task-specific labels. Most prior work addresses acquisition for a single predetermined task, limiting scalability. To address this limitation, we formalize the meta-AFA problem, where the goal is to learn acquisition policies across various tasks. We introduce Learning-to-Measure (L2M), which consists of i) reliable uncertainty quantification over unseen tasks, and ii) an uncertainty-guided greedy feature acquisition agent that maximizes conditional mutual information. We demonstrate a sequence-modeling or autoregressive pre-training approach that underpins reliable uncertainty quantification for tasks with arbitrary missingness. L2M operates directly on datasets with retrospective missingness and performs the meta-AFA task in-context, eliminating per-task retraining. Across synthetic and real-world tabular benchmarks, L2M matches or surpasses task-specific baselines, particularly under scarce labels and high missingness.