Conformal Sets in Multiple-Choice Question Answering under Black-Box Settings with Provable Coverage Guarantees
This work addresses the unreliability of LLMs in high-risk domains like medical QA by providing a model-agnostic framework for uncertainty quantification, though it is incremental as it builds on conformal prediction for black-box settings.
The authors tackled the problem of unreliable LLMs in multiple-choice question answering by proposing a frequency-based uncertainty quantification method that ensures provable coverage guarantees, showing it outperforms logit-based methods in distinguishing correct predictions across six LLMs and four datasets.
Large Language Models (LLMs) have shown remarkable progress in multiple-choice question answering (MCQA), but their inherent unreliability, such as hallucination and overconfidence, limits their application in high-risk domains. To address this, we propose a frequency-based uncertainty quantification method under black-box settings, leveraging conformal prediction (CP) to ensure provable coverage guarantees. Our approach involves multiple independent samplings of the model's output distribution for each input, with the most frequent sample serving as a reference to calculate predictive entropy (PE). Experimental evaluations across six LLMs and four datasets (MedMCQA, MedQA, MMLU, MMLU-Pro) demonstrate that frequency-based PE outperforms logit-based PE in distinguishing between correct and incorrect predictions, as measured by AUROC. Furthermore, the method effectively controls the empirical miscoverage rate under user-specified risk levels, validating that sampling frequency can serve as a viable substitute for logit-based probabilities in black-box scenarios. This work provides a distribution-free model-agnostic framework for reliable uncertainty quantification in MCQA with guaranteed coverage, enhancing the trustworthiness of LLMs in practical applications.