Dynamic and Generalizable Process Reward Modeling
This addresses the need for more adaptable and generalizable reward modeling in guiding Large Language Models for complex tasks, representing a novel method rather than an incremental improvement.
The paper tackles the problem of Process Reward Models (PRMs) struggling with cross-domain generalization and static evaluation in complex scenarios by proposing DG-PRM, which uses a reward tree and Pareto dominance estimation to dynamically select fine-grained rewards, achieving significant performance boosts on benchmarks and demonstrating strong generalizability to out-of-distribution scenarios.
Process Reward Models (PRMs) are crucial for guiding Large Language Models (LLMs) in complex scenarios by providing dense reward signals. However, existing PRMs primarily rely on heuristic approaches, which struggle with cross-domain generalization. While LLM-as-judge has been proposed to provide generalized rewards, current research has focused mainly on feedback results, overlooking the meaningful guidance embedded within the text. Additionally, static and coarse-grained evaluation criteria struggle to adapt to complex process supervision. To tackle these challenges, we propose Dynamic and Generalizable Process Reward Modeling (DG-PRM), which features a reward tree to capture and store fine-grained, multi-dimensional reward criteria. DG-PRM dynamically selects reward signals for step-wise reward scoring. To handle multifaceted reward signals, we pioneeringly adopt Pareto dominance estimation to identify discriminative positive and negative pairs. Experimental results show that DG-PRM achieves stunning performance on prevailing benchmarks, significantly boosting model performance across tasks with dense rewards. Further analysis reveals that DG-PRM adapts well to out-of-distribution scenarios, demonstrating exceptional generalizability.