CoLD: Counterfactually-Guided Length Debiasing for Process Reward Models
This addresses a reliability issue in PRMs for multi-step reasoning in LLMs, particularly in mathematical problem solving, but it is incremental as it focuses on mitigating a specific bias rather than introducing a new paradigm.
The paper tackles the problem of length bias in Process Reward Models (PRMs), where longer reasoning steps receive higher scores regardless of semantic content, and proposes CoLD, a counterfactually-guided debiasing framework that reduces reward-length correlation and improves step selection accuracy on benchmarks like MATH500 and GSM-Plus.
Process Reward Models (PRMs) play a central role in evaluating and guiding multi-step reasoning in large language models (LLMs), especially for mathematical problem solving. However, we identify a pervasive length bias in existing PRMs: they tend to assign higher scores to longer reasoning steps, even when the semantic content and logical validity are unchanged. This bias undermines the reliability of reward predictions and leads to overly verbose outputs during inference. To address this issue, we propose CoLD(Counterfactually-Guided Length Debiasing), a unified framework that mitigates length bias through three components: an explicit length-penalty adjustment, a learned bias estimator trained to capture spurious length-related signals, and a joint training strategy that enforces length-invariance in reward predictions. Our approach is grounded in counterfactual reasoning and informed by causal graph analysis. Extensive experiments on MATH500 and GSM-Plus show that CoLD consistently reduces reward-length correlation, improves accuracy in step selection, and encourages more concise, logically valid reasoning. These results demonstrate the effectiveness and practicality of CoLD in improving the fidelity and robustness of PRMs.