Calibration Error for Decision Making
This addresses the need for reliable probability interpretations in decision-making, offering a more robust calibration metric with practical algorithmic improvements.
The paper tackles the problem of calibration error in predictions for decision-making by proposing a new metric, Calibration Decision Loss (CDL), which measures the maximum payoff improvement from calibration across all tasks, and shows it can be efficiently achieved with near-optimal O(log T / sqrt(T)) error, bypassing a lower bound for existing metrics.
Calibration allows predictions to be reliably interpreted as probabilities by decision makers. We propose a decision-theoretic calibration error, the Calibration Decision Loss (CDL), defined as the maximum improvement in decision payoff obtained by calibrating the predictions, where the maximum is over all payoff-bounded decision tasks. Vanishing CDL guarantees the payoff loss from miscalibration vanishes simultaneously for all downstream decision tasks. We show separations between CDL and existing calibration error metrics, including the most well-studied metric Expected Calibration Error (ECE). Our main technical contribution is a new efficient algorithm for online calibration that achieves near-optimal $O(\frac{\log T}{\sqrt{T}})$ expected CDL, bypassing the $Ω(T^{-0.472})$ lower bound for ECE by Qiao and Valiant (2021).