Can Multiple Responses from an LLM Reveal the Sources of Its Uncertainty?
This work addresses the challenge of unreliable LLM outputs for real-world applications by providing a method to diagnose uncertainty sources, though it is incremental as it builds on existing uncertainty quantification efforts.
The study tackled the problem of diagnosing the sources of uncertainty in large language models by analyzing patterns of disagreement among multiple generated responses, using an auxiliary LLM to identify causes like input ambiguity or knowledge gaps, and validated the framework on datasets including AmbigQA, OpenBookQA, and MMLU-Pro.
Large language models (LLMs) have delivered significant breakthroughs across diverse domains but can still produce unreliable or misleading outputs, posing critical challenges for real-world applications. While many recent studies focus on quantifying model uncertainty, relatively little work has been devoted to \textit{diagnosing the source of uncertainty}. In this study, we show that, when an LLM is uncertain, the patterns of disagreement among its multiple generated responses contain rich clues about the underlying cause of uncertainty. To illustrate this point, we collect multiple responses from a target LLM and employ an auxiliary LLM to analyze their patterns of disagreement. The auxiliary model is tasked to reason about the likely source of uncertainty, such as whether it stems from ambiguity in the input question, a lack of relevant knowledge, or both. In cases involving knowledge gaps, the auxiliary model also identifies the specific missing facts or concepts contributing to the uncertainty. In our experiment, we validate our framework on AmbigQA, OpenBookQA, and MMLU-Pro, confirming its generality in diagnosing distinct uncertainty sources. Such diagnosis shows the potential for relevant manual interventions that improve LLM performance and reliability.