On the Calibration of Multiclass Classification with Rejection
This work addresses the challenge of reliable multiclass classification with rejection for applications requiring high accuracy, though it is incremental as it extends existing binary methods to multiclass settings.
The paper tackles the problem of multiclass classification with rejection, where a classifier can abstain from predictions to avoid errors, by analyzing calibration conditions and proposing new rejection criteria for general loss functions, achieving theoretical guarantees for Bayes-optimal solutions.
We investigate the problem of multiclass classification with rejection, where a classifier can choose not to make a prediction to avoid critical misclassification. First, we consider an approach based on simultaneous training of a classifier and a rejector, which achieves the state-of-the-art performance in the binary case. We analyze this approach for the multiclass case and derive a general condition for calibration to the Bayes-optimal solution, which suggests that calibration is hard to achieve by general loss functions unlike the binary case. Next, we consider another traditional approach based on confidence scores, in which the existing work focuses on a specific class of losses. We propose rejection criteria for more general losses for this approach and guarantee calibration to the Bayes-optimal solution. Finally, we conduct experiments to validate the relevance of our theoretical findings.