From <Answer> to <Think>: Multidimensional Supervision of Reasoning Process for LLM Optimization
This addresses the problem of flawed reasoning and sparse rewards in LLM optimization for researchers and practitioners, offering an incremental improvement over existing process-level reward models.
The paper tackles the challenge of improving multi-step reasoning in Large Language Models by proposing a Dimension-level Reward Model (DRM) that supervises reasoning along confidence, relevance, and coherence dimensions, achieving consistent gains on in-distribution and out-of-distribution tasks like mathematics and question answering.
Improving the multi-step reasoning ability of Large Language Models (LLMs) is a critical yet challenging task. The dominant paradigm, outcome-supervised reinforcement learning (RLVR), rewards only correct final answers, often propagating flawed reasoning and suffering from sparse reward signals. While process-level reward models (PRMs) provide denser, step-by-step feedback, they lack generalizability and interpretability, requiring task-specific segmentation of the reasoning process. To this end, we propose the Dimension-level Reward Model (DRM), a new supervision framework that bridges the gap between these two approaches. DRM evaluates the quality of a reasoning process along three fundamental, complementary, and interpretable dimensions: Confidence for uncertainty calibration, Relevance for semantic alignment, and Coherence for logical consistency. Together, these dimensions capture aspects beyond final answer correctness and enable interpretable assessment without requiring ground truth answers. Experimental results show that DRM provides effective supervision signals, guides the optimization of LLMs and enhances their reasoning ability. In particular, DRM-supervised training achieves consistent gains on both in-distribution and out-of-distribution open-domain tasks, including mathematics, question answering, code execution, and puzzles. Our findings demonstrate that multidimensional supervision of the reasoning process can improve the generalized reasoning ability of LLMs beyond the training distribution.