OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning
This addresses the limitation of reward models in evaluating long-form agentic tasks for AI alignment, though it is incremental as it builds on existing reward modeling with tool augmentation.
The paper tackled the problem of reward models struggling with knowledge-intensive and long-form tasks by introducing OpenRM, a tool-augmented reward model that uses external tools for evidence-based judgment, resulting in substantial performance gains over existing approaches on multiple benchmarks and improved downstream LLM alignment.
Reward models (RMs) have become essential for aligning large language models (LLMs), serving as scalable proxies for human evaluation in both training and inference. However, existing RMs struggle on knowledge-intensive and long-form tasks, where evaluating correctness requires grounding beyond the model's internal knowledge. This limitation hinders them from reliably discriminating subtle quality differences, especially when external evidence is necessary. To address this, we introduce OpenRM, a tool-augmented long-form reward model that systematically judges open-ended responses by invoking external tools to gather relevant evidence. We train OpenRM with Group Relative Policy Optimization (GRPO) on over 27K synthesized pairwise examples generated through a controllable data synthesis framework. The training objective jointly supervises intermediate tool usage and final outcome accuracy, incentivizing our reward model to learn effective evidence-based judgment strategies. Extensive experiments on three newly-collected datasets and two widely-used benchmarks demonstrate that OpenRM substantially outperforms existing reward modeling approaches. As a further step, we integrate OpenRM into both inference-time response selection and training-time data selection. This yields consistent gains in downstream LLM alignment tasks, highlighting the potential of tool-augmented reward models for scaling reliable long-form evaluation.