Embedding Perturbation may Better Reflect the Uncertainty in LLM Reasoning
This work addresses the need for fine-grained uncertainty estimation in LLM reasoning to enable targeted interventions, though it is incremental as it builds on existing uncertainty quantification techniques.
The study tackled the problem of quantifying uncertainty in large language models (LLMs) during reasoning tasks by identifying that incorrect intermediate steps are sensitive to perturbations in token embeddings, and showed that this perturbation-based metric outperforms baseline methods like token probability and entropy in uncertainty quantification performance.
Large language Models (LLMs) have achieved significant breakthroughs across diverse domains; however, they can still produce unreliable or misleading outputs. For responsible LLM application, Uncertainty Quantification (UQ) techniques are used to estimate a model's uncertainty about its outputs, indicating the likelihood that those outputs may be problematic. For LLM reasoning tasks, it is essential to estimate the uncertainty not only for the final answer, but also for the intermediate steps of the reasoning, as this can enable more fine-grained and targeted interventions. In this study, we explore what UQ metrics better reflect the LLM's ``intermediate uncertainty''during reasoning. Our study reveals that an LLMs' incorrect reasoning steps tend to contain tokens which are highly sensitive to the perturbations on the preceding token embeddings. In this way, incorrect (uncertain) intermediate steps can be readily identified using this sensitivity score as guidance in practice. In our experiments, we show such perturbation-based metric achieves stronger uncertainty quantification performance compared with baseline methods such as token (generation) probability and token entropy. Besides, different from approaches that rely on multiple sampling, the perturbation-based metrics offer better simplicity and efficiency.