Decision from Suboptimal Classifiers: Excess Risk Pre- and Post-Calibration
This work addresses the practical challenge of making reliable decisions with suboptimal classifiers, offering tools to quantify and mitigate regret, which is incremental but useful for practitioners in machine learning and NLP.
The paper tackles the problem of excess risk in binary decision-making when using approximate posterior probabilities from probabilistic classifiers, providing analytical expressions and bounds for miscalibration-induced and grouping loss regret, and demonstrates on NLP experiments that these estimates help determine when advanced post-training is cost-effective.
Probabilistic classifiers are central for making informed decisions under uncertainty. Based on the maximum expected utility principle, optimal decision rules can be derived using the posterior class probabilities and misclassification costs. Yet, in practice only learned approximations of the oracle posterior probabilities are available. In this work, we quantify the excess risk (a.k.a. regret) incurred using approximate posterior probabilities in batch binary decision-making. We provide analytical expressions for miscalibration-induced regret ($R^{\mathrm{CL}}$), as well as tight and informative upper and lower bounds on the regret of calibrated classifiers ($R^{\mathrm{GL}}$). These expressions allow us to identify regimes where recalibration alone addresses most of the regret, and regimes where the regret is dominated by the grouping loss, which calls for post-training beyond recalibration. Crucially, both $R^{\mathrm{CL}}$ and $R^{\mathrm{GL}}$ can be estimated in practice using a calibration curve and a recent grouping loss estimator. On NLP experiments, we show that these quantities identify when the expected gain of more advanced post-training is worth the operational cost. Finally, we highlight the potential of multicalibration approaches as efficient alternatives to costlier fine-tuning approaches.