Unsupervised Transformer Pre-Training for Images: Self-Distillation, Mean Teachers, and Random Crops
This is an incremental survey that analyzes and compares existing methods for unsupervised transformer pre-training in images, targeting researchers in computer vision.
The paper surveys DINOv2, a self-supervised learning method that achieves state-of-the-art performance by surpassing weakly supervised methods like OpenCLIP on most benchmarks, focusing on its core ideas of multi-crop view augmentation and self-distillation with a mean teacher.
Recent advances in self-supervised learning (SSL) have made it possible to learn general-purpose visual features that capture both the high-level semantics and the fine-grained spatial structure of images. Most notably, the recent DINOv2 has established a new state of the art by surpassing weakly supervised methods (WSL) like OpenCLIP on most benchmarks. In this survey, we examine the core ideas behind its approach, multi-crop view augmentation and self-distillation with a mean teacher, and trace their development in previous work. We then compare the performance of DINO and DINOv2 with other SSL and WSL methods across various downstream tasks, and highlight some remarkable emergent properties of their learned features with transformer backbones. We conclude by briefly discussing DINOv2's limitations, its impact, and future research directions.