Confidence-Aware Alignment Makes Reasoning LLMs More Reliable
For LLM reasoning, this provides a scalable method to improve reliability without external verifiers or massive sampling.
CASPO aligns token-level confidence with step-wise logical correctness via iterative DPO, enabling dynamic pruning of uncertain reasoning branches. It surpasses tree-search baselines on AIME'24 and AIME'25 without reward-model data, improving reasoning reliability and efficiency.
Large reasoning models often reach correct answers through flawed intermediate steps, creating a gap between final accuracy and reasoning reliability. Existing alignment strategies address this with external verifiers or massive sampling, limiting scalability. In this work, we introduce CASPO (Confidence-Aware Step-wise Preference Optimization), a framework that aligns token-level confidence with step-wise logical correctness through iterative Direct Preference Optimization, without training a separate reward model. During inference, we propose Confidence-aware Thought (CaT), which leverages this calibrated confidence to dynamically prune uncertain reasoning branches with negligible O(V) latency. Experiments across ten benchmarks and multiple model families show that CASPO consistently improves reasoning reliability and inference efficiency. CASPO scales to Qwen3-8B-Base and surpasses tree-search baselines on AIME'24 and AIME'25 without using reward-model data. We also release a step-wise dataset with confidence annotations to support fine-grained analysis of reasoning reliability. Code is available at https://github.com/Thecommonirin/CASPO.