Masked Siamese ConvNets
This work addresses a domain-specific issue in self-supervised learning for vision tasks, offering incremental improvements for researchers and practitioners using ConvNets.
The paper tackles the problem of masked siamese networks not working well with ConvNets, proposing empirical designs that achieve competitive performance on low-shot image classification and outperform previous methods on object detection benchmarks.
Self-supervised learning has shown superior performances over supervised methods on various vision benchmarks. The siamese network, which encourages embeddings to be invariant to distortions, is one of the most successful self-supervised visual representation learning approaches. Among all the augmentation methods, masking is the most general and straightforward method that has the potential to be applied to all kinds of input and requires the least amount of domain knowledge. However, masked siamese networks require particular inductive bias and practically only work well with Vision Transformers. This work empirically studies the problems behind masked siamese networks with ConvNets. We propose several empirical designs to overcome these problems gradually. Our method performs competitively on low-shot image classification and outperforms previous methods on object detection benchmarks. We discuss several remaining issues and hope this work can provide useful data points for future general-purpose self-supervised learning.