PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models
This addresses the problem of evaluating nuanced error detection in reasoning processes for AI researchers, though it is incremental as it builds on existing PRM concepts.
The paper tackles the lack of systematic evaluation for Process-Level Reward Models (PRMs) by introducing PRMBench, a benchmark with 6,216 problems and 83,456 step-level labels, and finds significant weaknesses in 15 tested models.
Process-level Reward Models (PRMs) are crucial for complex reasoning and decision-making tasks, where each intermediate step plays an important role in the reasoning process. Since language models are prone to various types of errors during the reasoning process, PRMs are required to possess nuanced capabilities for detecting various implicit error types in real-world scenarios. However, current benchmarks primarily focus on step correctness, failing to evaluate PRMs' performance systematically. To address this gap, we introduce PRMBench, a process-level benchmark specifically designed to assess the fine-grained error detection capabilities of PRMs. PRMBench comprises 6,216 carefully designed problems and 83,456 step-level labels, evaluating models across multiple dimensions, including simplicity, soundness, and sensitivity. In our experiments on 15 models, spanning both open-source PRMs and closed-source large language models prompted as critic models, we uncover significant weaknesses in current PRMs. These findings underscore the challenges inherent in process-level evaluation and highlight key directions for future research. We hope PRMBench can be a robust bench for advancing research on PRM evaluation and development.