Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimization
For researchers improving post-training of LLMs, this work provides a principled uncertainty signal that addresses limitations of existing entropy-based methods.
The paper identifies two gaps in entropy-based uncertainty estimators for policy optimization in LLMs—anisotropic and calibration gaps—and proposes GCPO, a framework using geometry-aware measures and reward-based calibration. Experiments show it improves gradient tracking and post-training performance.
Post-training has become central to improving reasoning and alignment in large language models, where critic-free models enable scalable learning from model-generated outputs but lack principled mechanisms to distinguish informative from noisy signals. Recent approaches leverage response-level measures as uncertainty signals to regulate group-based optimization methods such as GRPO. Yet their empirical success remains unstable and unclear in how they influence optimization dynamics. In this paper, we provide, to our knowledge, the first principled formulation that interprets uncertainty signals as mechanisms for characterizing and regulating gradient variance and learning signal quality. Based on both empirical and theoretical analysis, we identify two critical gaps of current entropy-based estimators: The anisotropic gap and The calibration gap. Motivated by this analysis, we propose Geometric-aware Calibrated Policy Optimization (GCPO), a novel framework integrating geometry-aware measures to capture semantic disagreement with reward-based calibration to align uncertainty with learning signal strength. Experiments on multiple benchmarks show that our approach more faithfully tracks gradient variability and consistently improves post-training performance. Our results highlight the importance of designing uncertainty signals that are aligned with optimization dynamics, offering a principled perspective for robust post-training.