Position: Uncertainty Quantification in LLMs is Just Unsupervised Clustering
For researchers and practitioners deploying LLMs in high-stakes domains, this paper highlights a fundamental flaw in current UQ methods that could lead to overconfidence in incorrect outputs.
The paper argues that current uncertainty quantification (UQ) methods for LLMs are essentially unsupervised clustering algorithms that measure internal consistency rather than external correctness, leading to a false sense of safety. It identifies three pathologies (hyperparameter sensitivity, conflation of stability with truth, lack of ground truth) and advocates for a paradigm shift toward truth-anchored UQ.
Uncertainty Quantification (UQ) is widely regarded as the primary safeguard for deploying Large Language Models (LLMs) in high-stakes domains. However, we argue that the field suffers from a category error: mainstream UQ methods for LLMs are just unsupervised clustering algorithms. We demonstrate that most current approaches inherently quantify the internal consistency of the model's generations rather than their external correctness. Consequently, current methods are fundamentally blind to factual reality and fail to detect ``confident hallucinations,'' where models exhibit high confidence in stable but incorrect answers. Therefore, the current UQ methods may create a deceptive sense of safety when deploying the models with uncertainty. In detail, we identify three critical pathologies resulting from this dependence on internal state: a hyperparameter sensitivity crisis that renders deployment unsafe, an internal evaluation cycle that conflates stability with truth, and a fundamental lack of ground truth that forces reliance on unstable proxy metrics to evaluate uncertainty. To resolve this impasse, we advocate for a paradigm shift to UQ and outline a roadmap for the research community to adopt better evaluation metrics and settings, implement mechanism changes for native uncertainty, and anchor verification in objective truth, ensuring that model confidence serves as a reliable proxy for reality.