UAlign: Leveraging Uncertainty Estimations for Factuality Alignment on Large Language Models
This work addresses the issue of factual inaccuracies in LLMs for users relying on them for knowledge-based tasks, representing an incremental improvement by integrating uncertainty estimations into existing alignment methods.
The paper tackles the problem of Large Language Models (LLMs) struggling to accurately express factual knowledge, especially when knowledge boundaries are ambiguous, by proposing the UAlign framework that leverages uncertainty estimations to improve factuality alignment, resulting in significant enhancements in confidently answering known questions and refusing unknown ones on in-domain and out-domain tasks.
Despite demonstrating impressive capabilities, Large Language Models (LLMs) still often struggle to accurately express the factual knowledge they possess, especially in cases where the LLMs' knowledge boundaries are ambiguous. To improve LLMs' factual expressions, we propose the UAlign framework, which leverages Uncertainty estimations to represent knowledge boundaries, and then explicitly incorporates these representations as input features into prompts for LLMs to Align with factual knowledge. First, we prepare the dataset on knowledge question-answering (QA) samples by calculating two uncertainty estimations, including confidence score and semantic entropy, to represent the knowledge boundaries for LLMs. Subsequently, using the prepared dataset, we train a reward model that incorporates uncertainty estimations and then employ the Proximal Policy Optimization (PPO) algorithm for factuality alignment on LLMs. Experimental results indicate that, by integrating uncertainty representations in LLM alignment, the proposed UAlign can significantly enhance the LLMs' capacities to confidently answer known questions and refuse unknown questions on both in-domain and out-of-domain tasks, showing reliability improvements and good generalizability over various prompt- and training-based baselines.