Hybrid Reward Normalization for Process-supervised Non-verifiable Agentic Tasks
This addresses the problem of training LLMs for complex reasoning tasks where intermediate steps cannot be easily verified, which is important for developers of AI agents.
The paper tackles the challenge of providing effective reinforcement learning supervision for LLMs performing complex agentic tasks with non-verifiable intermediate steps, by introducing Principle Process Reward (PPR) which unifies principled step-level assessment with outcome verification and a reward normalization strategy. The approach achieves state-of-the-art performance across multiple benchmarks, demonstrating robustness and generalization.
Large Language Models (LLMs) increasingly rely on external tools such as search engines to solve complex agentic tasks that require reasoning and external knowledge retrieval. Recently, reinforcement learning with verifiable rewards (RLVR) has demonstrated its effectiveness in advancing capabilities of LLMs by rewarding the final answers via outcome rewards. While straightforward to supervise, outcome rewards only provide sparse signals and delayed feedback, which limits their effectiveness on long trajectories. Process rewards address this by evaluating intermediate steps, providing fine-grained supervision and encouraging grounded problem solving. However, it is notoriously hard to annotate step-wise labels, especially in non-verifiable process without "golden" answers. Furthermore, step-wise judgment requires the balance between local quality with contribution to the final outcome, as optimizing towards higher process reward may not always align with better final outcomes. To address the above challenges, we introduce Principle Process Reward (PPR), an RL approach that unifies principled step-level assessment and outcome verification. We train a principle-based reward model to improve the transparency and reliability of process evaluation, and further introduce a Reward Normalization (ReNorm) strategy to calibrate outcome and process rewards. Experiment results show that PPR achieves state-of-the-art performance across a wide range of benchmarks, demonstrating its impressive robustness and generalization. Our code and model collection is available in this link.