Conformal Prediction for Long-Tailed Classification
This addresses the challenge of reliable uncertainty quantification in real-world applications like plant identification, where rare classes are often neglected, offering a practical solution for users needing verifiable predictions.
The paper tackles the problem of generating useful prediction sets for long-tailed classification, where existing methods force a trade-off between coverage and set size, by proposing new conformal prediction methods that smoothly trade off between set size and class-conditional coverage, achieving improved performance on datasets like Pl@ntNet-300K and iNaturalist-2018.
Many real-world classification problems, such as plant identification, have extremely long-tailed class distributions. In order for prediction sets to be useful in such settings, they should (i) provide good class-conditional coverage, ensuring that rare classes are not systematically omitted from the prediction sets, and (ii) be a reasonable size, allowing users to easily verify candidate labels. Unfortunately, existing conformal prediction methods, when applied to the long-tailed setting, force practitioners to make a binary choice between small sets with poor class-conditional coverage or sets with very good class-conditional coverage but that are extremely large. We propose methods with guaranteed marginal coverage that smoothly trade off between set size and class-conditional coverage. First, we introduce a new conformal score function called prevalence-adjusted softmax that targets macro-coverage, a relaxed notion of class-conditional coverage. Second, we propose a new procedure that interpolates between marginal and class-conditional conformal prediction by linearly interpolating their conformal score thresholds. We demonstrate our methods on Pl@ntNet-300K and iNaturalist-2018, two long-tailed image datasets with 1,081 and 8,142 classes, respectively.