Conformal Prediction of Classifiers with Many Classes based on Noisy Labels
This addresses uncertainty quantification for classification systems in noisy-label scenarios, which is incremental but important for practical applications.
The paper tackles the problem of calibrating conformal prediction thresholds when only noisy labels are available, and shows that their Noise-Aware Conformal Prediction method achieves finite sample coverage guarantees even with many classes.
Conformal Prediction (CP) controls the prediction uncertainty of classification systems by producing a small prediction set, ensuring a predetermined probability that the true class lies within this set. This is commonly done by defining a score, based on the model predictions, and setting a threshold on this score using a validation set. In this study, we address the problem of CP calibration when we only have access to a calibration set with noisy labels. We show how we can estimate the noise-free conformal threshold based on the noisy labeled data. We derive a finite sample coverage guarantee for uniform noise that remains effective even in tasks with a large number of classes. We dub our approach Noise-Aware Conformal Prediction (NACP). We illustrate the performance of the proposed results on several standard image classification datasets with a large number of classes.