GENUINE: Graph Enhanced Multi-level Uncertainty Estimation for Large Language Models
This work addresses the reliability of LLMs in high-stakes applications by enhancing uncertainty estimation, representing an incremental improvement through novel graph-based modeling.
The paper tackles the problem of uncertainty estimation in Large Language Models (LLMs) by proposing GENUINE, a structure-aware framework that uses dependency parse trees and hierarchical graph pooling to improve quantification, achieving up to 29% higher AUROC and over 15% reduction in calibration errors compared to existing methods.
Uncertainty estimation is essential for enhancing the reliability of Large Language Models (LLMs), particularly in high-stakes applications. Existing methods often overlook semantic dependencies, relying on token-level probability measures that fail to capture structural relationships within the generated text. We propose GENUINE: Graph ENhanced mUlti-level uncertaINty Estimation for Large Language Models, a structure-aware framework that leverages dependency parse trees and hierarchical graph pooling to refine uncertainty quantification. By incorporating supervised learning, GENUINE effectively models semantic and structural relationships, improving confidence assessments. Extensive experiments across NLP tasks show that GENUINE achieves up to 29% higher AUROC than semantic entropy-based approaches and reduces calibration errors by over 15%, demonstrating the effectiveness of graph-based uncertainty modeling. The code is available at https://github.com/ODYSSEYWT/GUQ.