Towards Harmonized Uncertainty Estimation for Large Language Models
This work addresses the need for reliable uncertainty quantification in LLMs to support trustworthy deployment, representing an incremental improvement over prior methods.
The paper tackles the problem of harmonizing uncertainty estimation for large language models by proposing CUE, a lightweight corrector method that improves scores by up to 60% over existing approaches.
To facilitate robust and trustworthy deployment of large language models (LLMs), it is essential to quantify the reliability of their generations through uncertainty estimation. While recent efforts have made significant advancements by leveraging the internal logic and linguistic features of LLMs to estimate uncertainty scores, our empirical analysis highlights the pitfalls of these methods to strike a harmonized estimation between indication, balance, and calibration, which hinders their broader capability for accurate uncertainty estimation. To address this challenge, we propose CUE (Corrector for Uncertainty Estimation): A straightforward yet effective method that employs a lightweight model trained on data aligned with the target LLM's performance to adjust uncertainty scores. Comprehensive experiments across diverse models and tasks demonstrate its effectiveness, which achieves consistent improvements of up to 60% over existing methods.