Conformal Risk Control for Non-Monotonic Losses
This work addresses the need for reliable risk control in machine learning applications with complex, non-monotonic loss functions, though it is incremental by building on existing conformal prediction methods.
The paper tackled the problem of extending conformal risk control to non-monotonic losses with multidimensional parameters, providing guarantees that depend on algorithm stability and applying it to tasks like selective image classification and multigroup debiasing.
Conformal risk control is an extension of conformal prediction for controlling risk functions beyond miscoverage. The original algorithm controls the expected value of a loss that is monotonic in a one-dimensional parameter. Here, we present risk control guarantees for generic algorithms applied to possibly non-monotonic losses with multidimensional parameters. The guarantees depend on the stability of the algorithm -- unstable algorithms have looser guarantees. We give applications of this technique to selective image classification, FDR and IOU control of tumor segmentations, and multigroup debiasing of recidivism predictions across overlapping race and sex groups using empirical risk minimization.