Few-Shot Calibration of Set Predictors via Meta-Learned Cross-Validation-Based Conformal Prediction
This work addresses the issue of uninformative uncertainty quantification in machine learning for applications requiring reliable set predictions with limited data, representing an incremental improvement over existing methods.
The paper tackles the problem of large predicted set sizes in conformal prediction under limited data by introducing a meta-learning solution called meta-XB, which reduces set size while preserving per-task calibration guarantees and is extended to adaptive non-conformal scores for improved calibration.
Conventional frequentist learning is known to yield poorly calibrated models that fail to reliably quantify the uncertainty of their decisions. Bayesian learning can improve calibration, but formal guarantees apply only under restrictive assumptions about correct model specification. Conformal prediction (CP) offers a general framework for the design of set predictors with calibration guarantees that hold regardless of the underlying data generation mechanism. However, when training data are limited, CP tends to produce large, and hence uninformative, predicted sets. This paper introduces a novel meta-learning solution that aims at reducing the set prediction size. Unlike prior work, the proposed meta-learning scheme, referred to as meta-XB, (i) builds on cross-validation-based CP, rather than the less efficient validation-based CP; and (ii) preserves formal per-task calibration guarantees, rather than less stringent task-marginal guarantees. Finally, meta-XB is extended to adaptive non-conformal scores, which are shown empirically to further enhance marginal per-input calibration.