Semantic Graph Consistency: Going Beyond Patches for Regularizing Self-Supervised Vision Transformers
This work addresses a specific bottleneck in self-supervised learning for vision transformers, offering incremental improvements in representation quality for computer vision tasks.
The paper tackles the underutilization of patch tokens in self-supervised vision transformers by introducing a Semantic Graph Consistency module that regularizes learning through graph-based message passing, resulting in a 5-10% performance increase in linear evaluation with limited labeled data.
Self-supervised learning (SSL) with vision transformers (ViTs) has proven effective for representation learning as demonstrated by the impressive performance on various downstream tasks. Despite these successes, existing ViT-based SSL architectures do not fully exploit the ViT backbone, particularly the patch tokens of the ViT. In this paper, we introduce a novel Semantic Graph Consistency (SGC) module to regularize ViT-based SSL methods and leverage patch tokens effectively. We reconceptualize images as graphs, with image patches as nodes and infuse relational inductive biases by explicit message passing using Graph Neural Networks into the SSL framework. Our SGC loss acts as a regularizer, leveraging the underexploited patch tokens of ViTs to construct a graph and enforcing consistency between graph features across multiple views of an image. Extensive experiments on various datasets including ImageNet, RESISC and Food-101 show that our approach significantly improves the quality of learned representations, resulting in a 5-10\% increase in performance when limited labeled data is used for linear evaluation. These experiments coupled with a comprehensive set of ablations demonstrate the promise of our approach in various settings.