Seed the Views: Hierarchical Semantic Alignment for Contrastive Representation Learning
This work improves representation learning for self-supervised learning methods, particularly for those based on contrastive learning, by enhancing the handling of semantically similar images.
The paper addresses the suboptimality of distinguishing semantically similar images in contrastive learning by proposing a hierarchical semantic alignment strategy. This method, CsMl, extends contrastive loss to allow multiple positives per anchor, achieving a 76.6% top-1 accuracy with linear evaluation using ResNet-50, and 66.7% and 75.1% top-1 accuracy with 1% and 10% labels, respectively, setting new state-of-the-art.
Self-supervised learning based on instance discrimination has shown remarkable progress. In particular, contrastive learning, which regards each image as well as its augmentations as an individual class and tries to distinguish them from all other images, has been verified effective for representation learning. However, pushing away two images that are de facto similar is suboptimal for general representation. In this paper, we propose a hierarchical semantic alignment strategy via expanding the views generated by a single image to \textbf{Cross-samples and Multi-level} representation, and models the invariance to semantically similar images in a hierarchical way. This is achieved by extending the contrastive loss to allow for multiple positives per anchor, and explicitly pulling semantically similar images/patches together at different layers of the network. Our method, termed as CsMl, has the ability to integrate multi-level visual representations across samples in a robust way. CsMl is applicable to current contrastive learning based methods and consistently improves the performance. Notably, using the moco as an instantiation, CsMl achieves a \textbf{76.6\% }top-1 accuracy with linear evaluation using ResNet-50 as backbone, and \textbf{66.7\%} and \textbf{75.1\%} top-1 accuracy with only 1\% and 10\% labels, respectively. \textbf{All these numbers set the new state-of-the-art.}