LLMs are Overconfident: Evaluating Confidence Interval Calibration with FermiEval
This addresses the issue of unreliable uncertainty quantification in LLMs for users relying on their numerical outputs, though it is incremental as it builds on existing calibration methods.
The paper tackled the problem of large language models (LLMs) being systematically overconfident in their numerical estimations, finding that nominal 99% confidence intervals only covered the true answer 65% of the time on average. With a conformal prediction approach, they achieved accurate 99% observed coverage and reduced the Winkler interval score by 54%.
Large language models (LLMs) excel at numerical estimation but struggle to correctly quantify uncertainty. We study how well LLMs construct confidence intervals around their own answers and find that they are systematically overconfident. To evaluate this behavior, we introduce FermiEval, a benchmark of Fermi-style estimation questions with a rigorous scoring rule for confidence interval coverage and sharpness. Across several modern models, nominal 99\% intervals cover the true answer only 65\% of the time on average. With a conformal prediction based approach that adjusts the intervals, we obtain accurate 99\% observed coverage, and the Winkler interval score decreases by 54\%. We also propose direct log-probability elicitation and quantile adjustment methods, which further reduce overconfidence at high confidence levels. Finally, we develop a perception-tunnel theory explaining why LLMs exhibit overconfidence: when reasoning under uncertainty, they act as if sampling from a truncated region of their inferred distribution, neglecting its tails.