Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learning
This addresses the need for reliable confidence estimates in LLMs used for decision-making, enabling better trust and fallback mechanisms, though it is incremental as it builds on existing fine-tuning paradigms.
The paper tackles the problem of overconfidence in decision-making LLMs fine-tuned with reinforcement learning, showing that RLVR improves accuracy but leads to poor calibration, while SFT offers better calibration with lower performance. The proposed calibration-aware reinforcement learning method reduces ECE scores by up to 9 points while maintaining accuracy.
Large language models (LLMs) are increasingly deployed in decision-making tasks, where not only accuracy but also reliable confidence estimates are essential. Well-calibrated confidence enables downstream systems to decide when to trust a model and when to defer to fallback mechanisms. In this work, we conduct a systematic study of calibration in two widely used fine-tuning paradigms: supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). We show that while RLVR improves task performance, it produces extremely overconfident models, whereas SFT yields substantially better calibration, even under distribution shift, though with smaller performance gains. Through targeted experiments, we diagnose RLVR's failure, showing that decision tokens act as extraction steps of the decision in reasoning traces and do not carry confidence information, which prevents reinforcement learning from surfacing calibrated alternatives. Based on this insight, we propose a calibration-aware reinforcement learning formulation that directly adjusts decision-token probabilities. Our method preserves RLVR's accuracy level while mitigating overconfidence, reducing ECE scores up to 9 points.