A Theoretical Framework for Preventing Class Collapse in Supervised Contrastive Learning
This work addresses a specific challenge in representation learning for practitioners, offering incremental improvements through theoretical insights.
The paper tackles the problem of class collapse in supervised contrastive learning by introducing a theoretical framework, SSEM, to analyze embedding structures and provide hyperparameter guidelines, with empirical validation on synthetic and real-world datasets.
Supervised contrastive learning (SupCL) has emerged as a prominent approach in representation learning, leveraging both supervised and self-supervised losses. However, achieving an optimal balance between these losses is challenging; failing to do so can lead to class collapse, reducing discrimination among individual embeddings in the same class. In this paper, we present theoretically grounded guidelines for SupCL to prevent class collapse in learned representations. Specifically, we introduce the Simplex-to-Simplex Embedding Model (SSEM), a theoretical framework that models various embedding structures, including all embeddings that minimize the supervised contrastive loss. Through SSEM, we analyze how hyperparameters affect learned representations, offering practical guidelines for hyperparameter selection to mitigate the risk of class collapse. Our theoretical findings are supported by empirical results across synthetic and real-world datasets.