Reward Modeling from Natural Language Human Feedback
This addresses a critical issue in reinforcement learning for AI systems by improving reward modeling accuracy, though it appears incremental as it builds on existing GRM and RLVR frameworks.
The paper tackles the problem of Generative Reward Models (GRMs) being susceptible to guessing correct outcomes without sound critiques in binary classification tasks, which introduces noise into reward signals and impairs reinforcement learning. The result is that their proposed method, Reward Modeling from Natural Language Human Feedback (RM-NLHF), consistently outperforms state-of-the-art GRMs trained with outcome-only reward on multiple benchmarks.
Reinforcement Learning with Verifiable reward (RLVR) on preference data has become the mainstream approach for training Generative Reward Models (GRMs). Typically in pairwise rewarding tasks, GRMs generate reasoning chains ending with critiques and preference labels, and RLVR then relies on the correctness of the preference labels as the training reward. However, in this paper, we demonstrate that such binary classification tasks make GRMs susceptible to guessing correct outcomes without sound critiques. Consequently, these spurious successes introduce substantial noise into the reward signal, thereby impairing the effectiveness of reinforcement learning. To address this issue, we propose Reward Modeling from Natural Language Human Feedback (RM-NLHF), which leverages natural language feedback to obtain process reward signals, thereby mitigating the problem of limited solution space inherent in binary tasks. Specifically, we compute the similarity between GRM-generated and human critiques as the training reward, which provides more accurate reward signals than outcome-only supervision. Additionally, considering that human critiques are difficult to scale up, we introduce Meta Reward Model (MetaRM) which learns to predict process reward from datasets with human critiques and then generalizes to data without human critiques. Experiments on multiple benchmarks demonstrate that our method consistently outperforms state-of-the-art GRMs trained with outcome-only reward, confirming the superiority of integrating natural language over binary human feedback as supervision.