Confidence as a Reward: Transforming LLMs into Reward Models
This work addresses the problem of reducing data and training costs for reward models in AI, offering a simple, effective solution for close-ended tasks, though it is incremental as it builds on prior ideas about confidence and training-free methods.
The paper tackles the challenge of enhancing large language models' reasoning without costly training by proposing Confidence-as-a-Reward (CRew), a training-free method that uses token-level confidence as a reward proxy, and shows it outperforms existing training-free approaches and most trained reward models on mathematical reasoning benchmarks like MATH500 and RewardMATH.
Reward models can significantly enhance the reasoning capabilities of large language models (LLMs), but they typically require extensive curated data and costly training. To mitigate these challenges, training-free approaches such as LLM-as-a-Judge leverage the intrinsic reasoning abilities of LLMs to evaluate responses, achieving promising results. Recent works have also indicated that model confidence can serve effectively as a reward metric, distinguishing between chain-of-thought (CoT) and non-CoT paths. However, the concept of using confidence as a reward has not been comprehensively studied. In this work, we systematically investigate Confidence-as-a-Reward (CRew), a simple yet powerful training-free method that utilizes token-level confidence in the model's final answers as a proxy for reward, especially suitable for close-ended tasks. Through extensive experiments on mathematical reasoning tasks, we demonstrate that CRew outperforms existing training-free reward approaches on the MATH500 and RewardMATH benchmarks, and even surpasses most trained reward models. We further identify a strong correlation between CRew scores and the actual reasoning performance of the model. Additionally, we find that CRew can effectively filter high-quality training data. Building upon these insights, we propose CRew-DPO, a training strategy that constructs preference data from confidence scores combined with correctness signals. Finetuning with CRew-DPO further enhances the model's judging capabilities and consistently outperforms existing self-training methods.