Beyond [cls]: Exploring the true potential of Masked Image Modeling representations
This addresses the practical limitation of MIM for users who cannot afford fine-tuning due to data, GPU, or knowledge constraints, offering an incremental improvement in representation usage.
The paper tackled the problem of poor out-of-the-box performance in Masked Image Modeling (MIM) for self-supervised learning, showing that uniform attention distribution leads to ineffective aggregation by the [cls] token, and proposed Selective Aggregation to improve performance significantly.
Masked Image Modeling (MIM) has emerged as a promising approach for Self-Supervised Learning (SSL) of visual representations. However, the out-of-the-box performance of MIMs is typically inferior to competing approaches. Most users cannot afford fine-tuning due to the need for large amounts of data, high GPU consumption, and specialized user knowledge. Therefore, the practical use of MIM representations is limited. In this paper we ask what is the reason for the poor out-of-the-box performance of MIMs. Is it due to weaker features produced by MIM models, or is it due to suboptimal usage? Through detailed analysis, we show that attention in MIMs is spread almost uniformly over many patches, leading to ineffective aggregation by the [cls] token. Based on this insight, we propose Selective Aggregation to better capture the rich semantic information retained in patch tokens, which significantly improves the out-of-the-box performance of MIM.