Downstream Task Guided Masking Learning in Masked Autoencoders Using Multi-Level Optimization
This addresses the problem of suboptimal visual representations for downstream tasks in self-supervised learning, though it appears incremental as it builds on existing MAE methods.
The paper tackles the limitation of Masked Autoencoder (MAE) in uniformly masking image patches by introducing MLO-MAE, which learns an optimal masking strategy using downstream task feedback, resulting in significant improvements across diverse datasets and tasks.
Masked Autoencoder (MAE) is a notable method for self-supervised pretraining in visual representation learning. It operates by randomly masking image patches and reconstructing these masked patches using the unmasked ones. A key limitation of MAE lies in its disregard for the varying informativeness of different patches, as it uniformly selects patches to mask. To overcome this, some approaches propose masking based on patch informativeness. However, these methods often do not consider the specific requirements of downstream tasks, potentially leading to suboptimal representations for these tasks. In response, we introduce the Multi-level Optimized Mask Autoencoder (MLO-MAE), a novel framework that leverages end-to-end feedback from downstream tasks to learn an optimal masking strategy during pretraining. Our experimental findings highlight MLO-MAE's significant advancements in visual representation learning. Compared to existing methods, it demonstrates remarkable improvements across diverse datasets and tasks, showcasing its adaptability and efficiency.