Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback
This addresses the challenge of aligning LLMs with human values more reliably, though it appears incremental as it builds on existing RLHF methods by adding uncertainty quantification.
The paper tackles the problem of unreliable reward learning in LLM alignment due to heterogeneous human feedback by developing a heterogeneous preference framework that models latent rewards and human rationality, enabling uncertainty quantification with theoretical guarantees and practical applications in statistical comparisons and policy frameworks.
We study estimation and statistical inference for reward models used in aligning large language models (LLMs). A key component of LLM alignment is reinforcement learning from human feedback (RLHF), where humans compare pairs of model-generated answers and their preferences are used to train a reward model. However, human feedback is inherently heterogeneous, creating significant challenges for reliable reward learning. To address this, we adopt a heterogeneous preference framework that jointly models the latent reward of answers and human rationality. This leads to a challenging biconvex optimization problem, which we solve via an alternating gradient descent algorithm. We establish theoretical guarantees for the resulting estimator, including its convergence and asymptotic distribution. These results enable the construction of confidence intervals for reward estimates. Leveraging these uncertainty quantification results, we conduct valid statistical comparisons between rewards and incorporate uncertainty into the best-of-$N$ (BoN) policy framework. Extensive simulations demonstrate the effectiveness of our method, and applications to real LLM data highlight the practical value of accounting for uncertainty in reward modeling for LLM alignment.