A Closer Look at Self-Supervised Lightweight Vision Transformers
This work addresses the performance gap in self-supervised learning for lightweight ViTs, which is incremental as it builds on existing pre-training methods but provides new insights and improvements.
The paper tackles the problem of self-supervised pre-training for lightweight Vision Transformers (ViTs), finding that with proper pre-training, vanilla lightweight ViTs achieve performance comparable to state-of-the-art networks, challenging the belief that they are unsuitable for lightweight vision tasks.
Self-supervised learning on large-scale Vision Transformers (ViTs) as pre-training methods has achieved promising downstream performance. Yet, how much these pre-training paradigms promote lightweight ViTs' performance is considerably less studied. In this work, we develop and benchmark several self-supervised pre-training methods on image classification tasks and some downstream dense prediction tasks. We surprisingly find that if proper pre-training is adopted, even vanilla lightweight ViTs show comparable performance to previous SOTA networks with delicate architecture design. It breaks the recently popular conception that vanilla ViTs are not suitable for vision tasks in lightweight regimes. We also point out some defects of such pre-training, e.g., failing to benefit from large-scale pre-training data and showing inferior performance on data-insufficient downstream tasks. Furthermore, we analyze and clearly show the effect of such pre-training by analyzing the properties of the layer representation and attention maps for related models. Finally, based on the above analyses, a distillation strategy during pre-training is developed, which leads to further downstream performance improvement for MAE-based pre-training. Code is available at https://github.com/wangsr126/mae-lite.