Self-Guided Masked Autoencoder
This work addresses a fundamental gap in understanding MAE's learning mechanisms for computer vision researchers, offering an incremental improvement to enhance efficiency in downstream tasks.
The paper tackled the problem of understanding and improving Masked Autoencoder (MAE) for self-supervised representation learning by discovering that MAE learns pattern-based patch clustering early in pretraining, and proposed a self-guided masking method that significantly boosts learning without external models.
Masked Autoencoder (MAE) is a self-supervised approach for representation learning, widely applicable to a variety of downstream tasks in computer vision. In spite of its success, it is still not fully uncovered what and how MAE exactly learns. In this paper, with an in-depth analysis, we discover that MAE intrinsically learns pattern-based patch-level clustering from surprisingly early stages of pretraining. Upon this understanding, we propose self-guided masked autoencoder, which internally generates informed mask by utilizing its progress in patch clustering, substituting the naive random masking of the vanilla MAE. Our approach significantly boosts its learning process without relying on any external models or supplementary information, keeping the benefit of self-supervised nature of MAE intact. Comprehensive experiments on various downstream tasks verify the effectiveness of the proposed method.