Variance-aware Reward Modeling with Anchor Guidance
For researchers and practitioners in RLHF, this work resolves a fundamental limitation of Gaussian reward models when preferences diverge, enabling more accurate uncertainty estimation.
The paper addresses the non-identifiability problem in Gaussian reward models for pluralistic human preferences by introducing anchor-guided variance-aware reward modeling. The method improves reward modeling and downstream RLHF performance across simulation and four real-world datasets.
Standard Bradley--Terry (BT) reward models are limited when human preferences are pluralistic. Although soft preference labels preserve disagreement information, BT can only express it by shrinking reward margins. Gaussian reward models provide an alternative by jointly predicting a reward mean and a reward variance, but suffer from a fundamental non-identifiability from pairwise preferences alone. We propose Anchor-guided Variance-aware Reward Modeling, a framework that resolves this non-identifiability by augmenting preference data with two coarse response-level anchor labels. Building on this, we prove that two anchors are sufficient for identification, develop a joint training objective and establish a non-asymptotic convergence rate for both the estimated reward mean and variance functions. Across simulation studies and four real-world diverging-preference datasets, our method consistently improves reward modeling performance and downstream RLHF, including PPO training and best-of-$N$ selection.