Beyond ECE: Calibrated Size Ratio, Risk Assessment, and Confidence-Weighted Metrics
For machine learning practitioners needing reliable confidence calibration metrics, this work addresses a critical flaw in the widely used ECE metric.
The paper identifies that Expected Calibration Error (ECE) can be small even under large overconfidence risk, and proposes Calibrated Size Ratio (CSR) as an interpretable alternative, along with confidence-weighted metrics like cwAUC that capture calibration information. Experiments show CSR achieves near-perfect sensitivity and specificity across synthetic and real datasets.
Confidence calibration has been dominated by the Expected Calibration Error (ECE), a linear metric that counts calibration offset equally regardless of the confidence level at which it occurs. We show that ECE can remain small even under arbitrarily large overconfidence risk, so we propose Calibrated Size Ratio (CSR) instead, an interpretable metric that equals 1 under perfect calibration, from which we derive the risk probability $P_{\mathrm{risk}}$ that quantifies the statistical evidence for overconfidence. We further argue that overconfidence risk assessment must be complemented by a measure of discriminative value: whether the assigned confidences actively distinguish correct from incorrect predictions. We show that confidence-weighted accuracy $\mathrm{cwA}$ is the natural such complement, and that confidence-weighting extends to all standard classification metrics. In particular, we prove that the confidence-weighted AUC (cwAUC) captures the information about calibration while the classical AUC cannot. We validate the proposed indicators on several synthetic confidence distributions under multiple controlled calibration profiles and on fifteen real datasets with and without post-hoc calibration. Experiments demonstrate that CSR achieves near-perfect sensitivity and specificity across all tested conditions.