Calibrating conditional risk
This work addresses uncertainty quantification for machine learning practitioners, but it is incremental as it builds on existing calibration concepts.
The paper tackles the problem of calibrating conditional risk, which estimates a prediction model's expected loss given input features, showing it is equivalent to standard regression and connecting it to existing calibration problems. Empirically, it validates these findings in learning to defer frameworks, providing qualitative and quantitative assessments.
We introduce and study the problem of calibrating conditional risk, which involves estimating the expected loss of a prediction model conditional on input features. We analyze this problem in both classification and regression settings and show that it is fundamentally equivalent to a standard regression task. For classification settings, we further establish a connection between conditional risk calibration and individual/conditional probability calibration, and develop theoretical insights for the performance metric. This reveals that while conditional risk calibration is related to existing uncertainty quantification problems, it remains a distinct and standalone machine learning problem. Empirically, we validate our theoretical findings and demonstrate the practical implications of conditional risk calibration in the learning to defer (L2D) framework. Our systematic experiments provide both qualitative and quantitative assessments, offering guidance for future research in uncertainty-aware decision-making.