Process Reward Model with Q-Value Rankings
This work addresses the challenge of accurately modeling intermediate steps in reasoning tasks for AI systems, representing an incremental improvement over existing PRM methods.
The paper tackled the problem of suboptimal reward distribution in Process Reward Modeling (PRM) for complex reasoning tasks by introducing the Process Q-value Model (PQM), which redefines PRM as a Markov Decision Process and uses a comparative loss function to optimize Q-value rankings, resulting in outperformance over classification-based PRMs in empirical evaluations.
Process Reward Modeling (PRM) is critical for complex reasoning and decision-making tasks where the accuracy of intermediate steps significantly influences the overall outcome. Existing PRM approaches, primarily framed as classification problems, employ cross-entropy loss to independently evaluate each step's correctness. This method can lead to suboptimal reward distribution and does not adequately address the interdependencies among steps. To address these limitations, we introduce the Process Q-value Model (PQM), a novel framework that redefines PRM in the context of a Markov Decision Process. PQM optimizes Q-value rankings based on a novel comparative loss function, enhancing the model's ability to capture the intricate dynamics among sequential decisions. This approach provides a more granular and theoretically grounded methodology for process rewards. Our extensive empirical evaluations across various sampling policies, language model backbones, and multi-step reasoning benchmarks show that PQM outperforms classification-based PRMs. The effectiveness of the comparative loss function is highlighted in our comprehensive ablation studies, confirming PQM's practical efficacy and theoretical advantage.