How does Contrastive Learning Organize Images?
This work addresses a fundamental understanding gap in self-supervised learning for computer vision researchers, offering insights into contrastive learning's limitations and a practical improvement method.
The paper investigates how contrastive learning organizes images by analyzing its inductive biases from a clustering perspective, finding it creates locally dense clusters that can hinder linear classification accuracy, and proposes using a Graph Convolutional Network (GCN) classifier to mitigate this, boosting accuracy and reducing parameters.
Contrastive learning, a dominant self-supervised technique, emphasizes similarity in representations between augmentations of the same input and dissimilarity for different ones. Although low contrastive loss often correlates with high classification accuracy, recent studies challenge this direct relationship, spotlighting the crucial role of inductive biases. We delve into these biases from a clustering viewpoint, noting that contrastive learning creates locally dense clusters, contrasting the globally dense clusters from supervised learning. To capture this discrepancy, we introduce the "RLD (Relative Local Density)" metric. While this cluster property can hinder linear classification accuracy, leveraging a Graph Convolutional Network (GCN) based classifier mitigates this, boosting accuracy and reducing parameter requirements. The code is available \href{https://github.com/xsgxlz/How-does-Contrastive-Learning-Organize-Images/tree/main}{here}.