Rethinking Uncertainty Evaluation in Large Language Models
For researchers and practitioners using LLM confidence estimates, this work highlights a fundamental flaw in current evaluation and provides a new framework to measure probabilistic coherence.
The paper argues that calibration is insufficient for evaluating LLM confidence estimates, proposing three axes (structural coherence, faithfulness, usefulness) and C1 metrics. Experiments show widely used estimators violate these conditions despite appearing well-calibrated, e.g., models assign lower confidence to logically easier questions 31% of the time.
Calibration is the primary criterion for evaluating LLM confidence, but it is insufficient: it admits trivially incoherent estimators, depends on the evaluation distribution, and does not test the extent to which the estimation can be interpreted as a consistent, underlying probability function. What we actually need is for LLM confidence estimates to satisfy the conditions required of coherent probabilistic beliefs. We formalize these conditions along three axes (structural coherence, faithfulness, and usefulness) and operationalize them as the C1 metrics. Widely used estimators systematically violate these conditions despite appearing well-calibrated: models assign lower confidence to logically easier questions 31\% of the time, and common interventions reducing RMSCE leave structural violations unchanged, suggesting that calibration is orthogonal to probabilistic validity. RLHF and chain-of-thought improve usefulness metrics without restoring coherence. Our results show current LLM confidence estimates cannot be interpreted as coherent probabilities; our framework provides the tools to measure and close this gap.