Leveling Down in Computer Vision: Pareto Inefficiencies in Fair Deep Classifiers
This work addresses algorithmic fairness in computer vision, revealing inefficiencies in current methods and offering a novel solution, though it is incremental in improving specific fairness approaches.
The paper tackles the problem of fairness in computer vision classifiers, finding that existing fairness methods degrade performance across all groups, including disadvantaged ones, and proposes an adaptive augmentation strategy that improves performance for disadvantaged groups.
Algorithmic fairness is frequently motivated in terms of a trade-off in which overall performance is decreased so as to improve performance on disadvantaged groups where the algorithm would otherwise be less accurate. Contrary to this, we find that applying existing fairness approaches to computer vision improve fairness by degrading the performance of classifiers across all groups (with increased degradation on the best performing groups). Extending the bias-variance decomposition for classification to fairness, we theoretically explain why the majority of fairness classifiers designed for low capacity models should not be used in settings involving high-capacity models, a scenario common to computer vision. We corroborate this analysis with extensive experimental support that shows that many of the fairness heuristics used in computer vision also degrade performance on the most disadvantaged groups. Building on these insights, we propose an adaptive augmentation strategy that, uniquely, of all methods tested, improves performance for the disadvantaged groups.