Selective Classification Can Magnify Disparities Across Groups
This highlights a fairness problem in machine learning for practitioners using selective classification, as it can exacerbate group disparities, making it an incremental but important cautionary finding.
The paper finds that selective classification, which improves average accuracy by allowing models to abstain on uncertain predictions, can magnify accuracy disparities between groups, especially with spurious correlations, and may even decrease accuracy for some groups; it provides theoretical analysis and shows that training distributionally-robust models can mitigate this issue.
Selective classification, in which models can abstain on uncertain predictions, is a natural approach to improving accuracy in settings where errors are costly but abstentions are manageable. In this paper, we find that while selective classification can improve average accuracies, it can simultaneously magnify existing accuracy disparities between various groups within a population, especially in the presence of spurious correlations. We observe this behavior consistently across five vision and NLP datasets. Surprisingly, increasing abstentions can even decrease accuracies on some groups. To better understand this phenomenon, we study the margin distribution, which captures the model's confidences over all predictions. For symmetric margin distributions, we prove that whether selective classification monotonically improves or worsens accuracy is fully determined by the accuracy at full coverage (i.e., without any abstentions) and whether the distribution satisfies a property we call left-log-concavity. Our analysis also shows that selective classification tends to magnify full-coverage accuracy disparities. Motivated by our analysis, we train distributionally-robust models that achieve similar full-coverage accuracies across groups and show that selective classification uniformly improves each group on these models. Altogether, our results suggest that selective classification should be used with care and underscore the importance of training models to perform equally well across groups at full coverage.