CAOS: Conformal Aggregation of One-Shot Predictors
This addresses the need for reliable uncertainty estimation in one-shot learning scenarios, which is incremental as it builds on conformal prediction methods.
The paper tackled the problem of lacking principled uncertainty quantification in one-shot prediction with pretrained foundation models, and the result was that CAOS achieved valid marginal coverage while producing substantially smaller prediction sets than split conformal baselines in experiments on facial landmarking and text classification tasks.
One-shot prediction enables rapid adaptation of pretrained foundation models to new tasks using only one labeled example, but lacks principled uncertainty quantification. While conformal prediction provides finite-sample coverage guarantees, standard split conformal methods are inefficient in the one-shot setting due to data splitting and reliance on a single predictor. We propose Conformal Aggregation of One-Shot Predictors (CAOS), a conformal framework that adaptively aggregates multiple one-shot predictors and uses a leave-one-out calibration scheme to fully exploit scarce labeled data. Despite violating classical exchangeability assumptions, we prove that CAOS achieves valid marginal coverage using a monotonicity-based argument. Experiments on one-shot facial landmarking and RAFT text classification tasks show that CAOS produces substantially smaller prediction sets than split conformal baselines while maintaining reliable coverage.