An Efficient and Precise Training Data Construction Framework for Process-supervised Reward Model in Mathematical Reasoning
This work addresses the problem of inefficient and low-quality data construction for process-supervised reward models in mathematical reasoning, offering a practical solution for researchers and developers, though it is incremental as it builds on existing PRM methods.
The paper tackles the high cost and poor quality of constructing process-supervised reward model (PRM) training data for mathematical reasoning in LLMs by introducing EpicPRM, a framework that uses quantified contributions and adaptive binary search to efficiently create a 50k-step dataset, leading to significantly superior PRM performance compared to existing datasets.
Enhancing the mathematical reasoning capabilities of Large Language Models (LLMs) is of great scientific and practical significance. Researchers typically employ process-supervised reward models (PRMs) to guide the reasoning process, effectively improving the models' reasoning abilities. However, existing methods for constructing process supervision training data, such as manual annotation and per-step Monte Carlo estimation, are often costly or suffer from poor quality. To address these challenges, this paper introduces a framework called EpicPRM, which annotates each intermediate reasoning step based on its quantified contribution and uses an adaptive binary search algorithm to enhance both annotation precision and efficiency. Using this approach, we efficiently construct a high-quality process supervision training dataset named Epic50k, consisting of 50k annotated intermediate steps. Compared to other publicly available datasets, the PRM trained on Epic50k demonstrates significantly superior performance. Getting Epic50k at https://github.com/xiaolizh1/EpicPRM.