When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs
For researchers comparing LLM calibration, this work reveals that raw global metrics are unreliable and provides a fairer evaluation method.
The paper shows that global calibration metrics like ECE and Brier Score are confounded by accuracy differences when comparing LLMs, and proposes ACE, an accuracy-controlled evaluation framework. Using ACE, they find that many previously reported calibration advantages weaken or reverse after controlling for accuracy.
Calibration evaluates whether a model confidence aligns with its empirical accuracy. Existing studies often compare the calibration of different large language models using global calibration metrics such as Expected Calibration Error and Brier Score. We begin by showing, both theoretically and empirically, that such comparisons are confounded by differences in model accuracy. For fairer cross-model comparison, we then propose ACE, an accuracy-controlled evaluation framework with three complementary views: Instance-Aligned, Distribution-Aligned, and Candidate-Aligned calibration. Across multiple benchmarks, model families, and confidence elicitation methods, we use ACE to study two practically important comparison axes, small versus large models and thinking versus non-thinking models. We find that many previously reported calibration advantages under raw global metrics weaken substantially after accuracy control. We also find that ranking reversal is frequent: models favored by raw metrics often cease to be favored once accuracy is controlled. Our results show that raw global calibration metrics are not robust for cross-model comparison, and that fair calibration comparison requires accuracy-aware evaluation.