Why Don't You Know? Evaluating the Impact of Uncertainty Sources on Uncertainty Quantification in LLMs
For researchers and practitioners deploying LLMs, this work highlights the need for UQ methods that account for multiple uncertainty sources, as current approaches can be misleading.
The paper studies how different sources of uncertainty (model knowledge gaps, output variability, input ambiguity) affect existing uncertainty quantification (UQ) methods for LLMs. It introduces a new dataset categorizing uncertainty sources and finds that UQ methods perform well only when uncertainty stems from model knowledge limitations, degrading otherwise.
As Large Language Models (LLMs) are increasingly deployed in real-world applications, reliable uncertainty quantification (UQ) becomes critical for safe and effective use. Most existing UQ approaches for language models aim to produce a single confidence score -- for example, estimating the probability that a model's answer is correct. However, uncertainty in natural language tasks arises from multiple distinct sources, including model knowledge gaps, output variability, and input ambiguity, which have different implications for system behavior and user interaction. In this work, we study how the source of uncertainty impacts the behavior and effectiveness of existing UQ methods. To enable controlled analysis, we introduce a new dataset that explicitly categorizes uncertainty sources, allowing systematic evaluation of UQ performance under each condition. Our experiments reveal that while many UQ methods perform well when uncertainty stems solely from model knowledge limitations, their performance degrades or becomes misleading when other sources are introduced. These findings highlight the need for uncertainty-aware methods that explicitly account for the source of uncertainty in large language models.