Improving Uncertainty Estimation through Semantically Diverse Language Generation
This addresses the issue of untrustworthy LLMs for society and industry, representing a strong specific gain in uncertainty estimation.
The paper tackles the problem of hallucinations in large language models by introducing Semantically Diverse Language Generation (SDLG) to quantify predictive uncertainty, showing it outperforms existing methods and is computationally efficient in question-answering tasks.
Large language models (LLMs) can suffer from hallucinations when generating text. These hallucinations impede various applications in society and industry by making LLMs untrustworthy. Current LLMs generate text in an autoregressive fashion by predicting and appending text tokens. When an LLM is uncertain about the semantic meaning of the next tokens to generate, it is likely to start hallucinating. Thus, it has been suggested that predictive uncertainty is one of the main causes of hallucinations. We introduce Semantically Diverse Language Generation (SDLG) to quantify predictive uncertainty in LLMs. SDLG steers the LLM to generate semantically diverse yet likely alternatives for an initially generated text. This approach provides a precise measure of aleatoric semantic uncertainty, detecting whether the initial text is likely to be hallucinated. Experiments on question-answering tasks demonstrate that SDLG consistently outperforms existing methods while being the most computationally efficient, setting a new standard for uncertainty estimation in LLMs.