Class-Conditional Conformal Prediction with Many Classes
This addresses the need for more reliable uncertainty quantification in classification tasks with many classes, such as image recognition, but is incremental as it builds on existing conformal prediction methods.
The paper tackles the problem of achieving class-conditional coverage in conformal prediction when there are many classes and limited data per class, proposing clustered conformal prediction that groups similar classes and shows improved coverage and set size metrics across four image datasets with up to 1000 classes.
Standard conformal prediction methods provide a marginal coverage guarantee, which means that for a random test point, the conformal prediction set contains the true label with a user-specified probability. In many classification problems, we would like to obtain a stronger guarantee--that for test points of a specific class, the prediction set contains the true label with the same user-chosen probability. For the latter goal, existing conformal prediction methods do not work well when there is a limited amount of labeled data per class, as is often the case in real applications where the number of classes is large. We propose a method called clustered conformal prediction that clusters together classes having "similar" conformal scores and performs conformal prediction at the cluster level. Based on empirical evaluation across four image data sets with many (up to 1000) classes, we find that clustered conformal typically outperforms existing methods in terms of class-conditional coverage and set size metrics.