COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees
This addresses the need for reliable uncertainty quantification in foundation models to reduce hallucinations, offering a method with formal guarantees for selective question answering, though it is incremental as it builds on existing conformal prediction frameworks.
The paper tackles the problem of uncertainty quantification in foundation models to mitigate hallucinations by proposing COIN, a framework that calibrates thresholds to select a single answer per question with provable false discovery rate (FDR) control, achieving significant increases in sample retention while maintaining robustness in risk control across tasks.
Uncertainty quantification (UQ) for foundation models is essential to identify and mitigate potential hallucinations in automatically generated text. However, heuristic UQ approaches lack formal guarantees for key metrics such as the false discovery rate (FDR) in selective prediction. Previous work adopts the split conformal prediction (SCP) framework to ensure desired coverage of admissible answers by constructing prediction sets, but these sets often contain incorrect candidates, limiting their practical utility. To address this, we propose COIN, an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question under user-specified FDR constraints. COIN estimates the empirical error rate on a calibration set and applies confidence interval methods such as Clopper-Pearson to establish a high-probability upper bound on the true error rate (i.e., FDR). This enables the selection of the largest uncertainty threshold that ensures FDR control on test data while significantly increasing sample retention. We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data across both general and multimodal text generation tasks. Furthermore, we show that employing alternative upper bound constructions and UQ strategies can further boost COIN's power performance, which underscores its extensibility and adaptability to diverse application scenarios.