Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization
This work addresses the challenge of accurately evaluating reward models for researchers and practitioners in reinforcement learning from human feedback, though it is incremental as it refines evaluation methods rather than introducing a new paradigm.
The paper tackles the problem that existing benchmarks for reward models in RLHF show weak correlation with optimized policy performance, and finds that constructing reliable benchmarks requires minimizing differences beyond correctness, using multiple comparisons across diverse responses, and sourcing from various models, while noting that high correlation with overoptimization can lower downstream performance.
Reward models (RMs) play a crucial role in reinforcement learning from human feedback (RLHF), aligning model behavior with human preferences. However, existing benchmarks for reward models show a weak correlation with the performance of optimized policies, suggesting that they fail to accurately assess the true capabilities of RMs. To bridge this gap, we explore several evaluation designs through the lens of reward overoptimization\textemdash a phenomenon that captures both how well the reward model aligns with human preferences and the dynamics of the learning signal it provides to the policy. The results highlight three key findings on how to construct a reliable benchmark: (i) it is important to minimize differences between chosen and rejected responses beyond correctness, (ii) evaluating reward models requires multiple comparisons across a wide range of chosen and rejected responses, and (iii) given that reward models encounter responses with diverse representations, responses should be sourced from a variety of models. However, we also observe that a extremely high correlation with degree of overoptimization leads to comparatively lower correlation with certain downstream performance. Thus, when designing a benchmark, it is desirable to use the degree of overoptimization as a useful tool, rather than the end goal.