PACR: Progressively Ascending Confidence Reward for LLM Reasoning
This work addresses slow exploration in LLM reasoning for AI researchers, offering an incremental improvement over existing RLVR methods.
The paper tackled the problem of sparse rewards in reinforcement learning for LLM reasoning by proposing PACR, a dense reward based on the model's evolving confidence in the correct answer, which accelerated exploration and improved performance on multiple benchmarks.
Reinforcement Learning with Verifiable Rewards (RLVR) has significantly improved LLM reasoning, but its sparse, outcome-based reward provides no guidance for intermediate steps, slowing exploration. We propose Progressively Ascending Confidence Reward (PACR), a dense, model-intrinsic reward computed directly from the model's evolving belief in the correct answer. PACR encodes the inductive bias that, along a well-formed reasoning trajectory, the probability of the ground-truth answer should have a generally ascending trend. We provide empirical and theoretical analysis validating that such an inductive bias constrains the exploration search space to regions richer in logically sound reasoning. We demonstrate that PACR accelerates exploration, reaches reward saturation with fewer trajectories, and yields improvements on multiple benchmarks. Our results suggest that dense, model-intrinsic shaping signals can make RLVR training more effective and reliable.