Domain-Shift-Aware Conformal Prediction for Large Language Models
This addresses the need for trustworthy uncertainty quantification in large language models for real-world deployment, representing an incremental improvement over existing conformal prediction techniques.
The paper tackles the problem of overconfident and factually incorrect outputs (hallucinations) in large language models under domain shift, proposing a Domain-Shift-Aware Conformal Prediction framework that delivers more reliable coverage than standard methods, as demonstrated on the MMLU benchmark.
Large language models have achieved impressive performance across diverse tasks. However, their tendency to produce overconfident and factually incorrect outputs, known as hallucinations, poses risks in real world applications. Conformal prediction provides finite-sample, distribution-free coverage guarantees, but standard conformal prediction breaks down under domain shift, often leading to under-coverage and unreliable prediction sets. We propose a new framework called Domain-Shift-Aware Conformal Prediction (DS-CP). Our framework adapts conformal prediction to large language models under domain shift, by systematically reweighting calibration samples based on their proximity to the test prompt, thereby preserving validity while enhancing adaptivity. Our theoretical analysis and experiments on the MMLU benchmark demonstrate that the proposed method delivers more reliable coverage than standard conformal prediction, especially under substantial distribution shifts, while maintaining efficiency. This provides a practical step toward trustworthy uncertainty quantification for large language models in real-world deployment.