Probability-Consistent Preference Optimization for Enhanced LLM Reasoning
This work addresses the challenge of enhancing reasoning accuracy in LLMs for applications like mathematical problem-solving, representing an incremental improvement over existing preference optimization techniques.
The paper tackles the problem of improving mathematical reasoning in large language models by addressing the neglect of internal logical coherence in current preference optimization methods, proposing Probability-Consistent Preference Optimization (PCPO) which uses dual metrics for answer correctness and token-level probability consistency, and shows it consistently outperforms existing approaches across various LLMs and benchmarks.
Recent advances in preference optimization have demonstrated significant potential for improving mathematical reasoning capabilities in large language models (LLMs). While current approaches leverage high-quality pairwise preference data through outcome-based criteria like answer correctness or consistency, they fundamentally neglect the internal logical coherence of responses. To overcome this, we propose Probability-Consistent Preference Optimization (PCPO), a novel framework that establishes dual quantitative metrics for preference selection: (1) surface-level answer correctness and (2) intrinsic token-level probability consistency across responses. Extensive experiments show that our PCPO consistently outperforms existing outcome-only criterion approaches across a diverse range of LLMs and benchmarks. Our code is publicly available at https://github.com/YunqiaoYang/PCPO.