Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilities
This addresses the problem of unreliable uncertainty quantification in LLMs for users needing credible outputs, though it is incremental as it builds on existing imprecise probability frameworks.
The paper tackled the mismatch between LLM behavior and classical uncertainty elicitation techniques by proposing prompt-based methods grounded in imprecise probabilities to capture higher-order uncertainty, demonstrating improved uncertainty reporting across diverse settings.
Despite the growing demand for eliciting uncertainty from large language models (LLMs), empirical evidence suggests that LLM behavior is not always adequately captured by the elicitation techniques developed under the classical probabilistic uncertainty framework. This mismatch leads to systematic failure modes, particularly in settings that involve ambiguous question-answering, in-context learning, and self-reflection. To address this, we propose novel prompt-based uncertainty elicitation techniques grounded in \emph{imprecise probabilities}, a principled framework for repesenting and eliciting higher-order uncertainty. Here, first-order uncertainty captures uncertainty over possible responses to a prompt, while second-order uncertainty (uncertainty about uncertainty) quantifies indeterminacy in the underlying probability model itself. We introduce general-purpose prompting and post-processing procedures to directly elicit and quantify both orders of uncertainty, and demonstrate their effectiveness across diverse settings. Our approach enables more faithful uncertainty reporting from LLMs, improving credibility and supporting downstream decision-making.