Better Language Model-Based Judging Reward Modeling through Scaling Comprehension Boundaries
This work addresses the challenge of building better reward models for reinforcement learning from AI feedback, representing an incremental advancement in the LM-based judging paradigm.
The paper tackles the problem of improving language model-based reward modeling by reframing it as a natural language inference task and scaling comprehension boundaries, resulting in ESFP-RM which delivers more stable and generalizable reward signals in RLHF and OOD scenarios compared to generative reward models.
The emergence of LM-based judging reward modeling, represented by generative reward models, has successfully made reinforcement learning from AI feedback (RLAIF) efficient and scalable. To further advance this paradigm, we propose a core insight: this form of reward modeling shares fundamental formal consistency with natural language inference (NLI), a core task in natural language understanding. This reframed perspective points to a key path for building superior reward models: scaling the model's comprehension boundaries. Pursuing this path, exploratory experiments on NLI tasks demonstrate that the slot prediction masked language models (MLMs) incorporating contextual explanations achieve significantly better performance compared to mainstream autoregressive models. Based on this key finding, we propose ESFP-RM, a two-stage LM-based judging reward model that utilizes an explanation based slot framework for prediction to fully leverage the advantages of MLMs. Extensive experiments demonstrate that in both reinforcement learning from human feedback (RLHF) and out-of-distribution (OOD) scenarios, the ESFP-RM framework delivers more stable and generalizable reward signals compared to generative reward models.