The Dynamic Duo of Collaborative Masking and Target for Advanced Masked Autoencoder Learning
This work addresses a specific bottleneck in self-supervised learning for computer vision, offering an incremental improvement over existing methods.
The paper tackles the problem of improving masked autoencoders for self-supervised vision representation learning by integrating collaborative masking and targets, resulting in a fine-tuning accuracy increase from 83.6% to 85.7% on ImageNet-1K with ViT-base.
Masked autoencoders (MAE) have recently succeeded in self-supervised vision representation learning. Previous work mainly applied custom-designed (e.g., random, block-wise) masking or teacher (e.g., CLIP)-guided masking and targets. However, they ignore the potential role of the self-training (student) model in giving feedback to the teacher for masking and targets. In this work, we present to integrate Collaborative Masking and Targets for boosting Masked AutoEncoders, namely CMT-MAE. Specifically, CMT-MAE leverages a simple collaborative masking mechanism through linear aggregation across attentions from both teacher and student models. We further propose using the output features from those two models as the collaborative target of the decoder. Our simple and effective framework pre-trained on ImageNet-1K achieves state-of-the-art linear probing and fine-tuning performance. In particular, using ViT-base, we improve the fine-tuning results of the vanilla MAE from 83.6% to 85.7%.