Step-level Value Preference Optimization for Mathematical Reasoning
This work addresses the limitation of overall preference annotations in complex reasoning tasks for AI researchers and practitioners, offering an incremental improvement over existing preference optimization methods.
The paper tackles the problem of fine-grained quality assessment in multi-step mathematical reasoning by introducing Step-level Value Preference Optimization (SVPO), which uses Monte Carlo Tree Search for step-level preference annotation and an explicit value model, achieving state-of-the-art performance on in-domain and out-of-domain benchmarks.
Direct Preference Optimization (DPO) using an implicit reward model has proven to be an effective alternative to reinforcement learning from human feedback (RLHF) for fine-tuning preference aligned large language models (LLMs). However, the overall preference annotations of responses do not fully capture the fine-grained quality of model outputs in complex multi-step reasoning tasks, such as mathematical reasoning. To address this limitation, we introduce a novel algorithm called Step-level Value Preference Optimization (SVPO). Our approach employs Monte Carlo Tree Search (MCTS) to automatically annotate step-level preferences for multi-step reasoning. Furthermore, from the perspective of learning-to-rank, we train an explicit value model to replicate the behavior of the implicit reward model, complementing standard preference optimization. This value model enables the LLM to generate higher reward responses with minimal cost during inference. Experimental results demonstrate that our method achieves state-of-the-art performance on both in-domain and out-of-domain mathematical reasoning benchmarks. Our code is available at \url{https://github.com/MARIO-Math-Reasoning/Super_MARIO}.