An Investigation of Representation and Allocation Harms in Contrastive Learning
This addresses fairness issues in self-supervised learning for AI practitioners, though it is incremental as it extends known supervised learning problems to CL.
The paper investigates how contrastive learning (CL) in self-supervised learning collapses representations of minority groups with majority groups, termed representation harm, and shows this leads to allocation harm in downstream tasks, with theoretical support from a stochastic block model.
The effect of underrepresentation on the performance of minority groups is known to be a serious problem in supervised learning settings; however, it has been underexplored so far in the context of self-supervised learning (SSL). In this paper, we demonstrate that contrastive learning (CL), a popular variant of SSL, tends to collapse representations of minority groups with certain majority groups. We refer to this phenomenon as representation harm and demonstrate it on image and text datasets using the corresponding popular CL methods. Furthermore, our causal mediation analysis of allocation harm on a downstream classification task reveals that representation harm is partly responsible for it, thus emphasizing the importance of studying and mitigating representation harm. Finally, we provide a theoretical explanation for representation harm using a stochastic block model that leads to a representational neural collapse in a contrastive learning setting.