Rethinking Random Masking in Self-Distillation on ViT
This addresses a specific technical issue in self-distillation for Vision Transformers, offering an incremental improvement to masking strategies.
The paper tackled the problem that random masking in self-distillation frameworks like DINO can eliminate critical semantic information, and found that applying random masking only to the student's global view while keeping other views clean yields more robust attention maps and enhances downstream performance on mini-ImageNet with DINO-Tiny.
Vision Transformers (ViTs) have demonstrated remarkable performance across a wide range of vision tasks. In particular, self-distillation frameworks such as DINO have contributed significantly to these advances. Within such frameworks, random masking is often utilized to improve training efficiency and introduce regularization. However, recent studies have raised concerns that indiscriminate random masking may inadvertently eliminate critical semantic information, motivating the development of more informed masking strategies. In this study, we explore the role of random masking in the self-distillation setting, focusing on the DINO framework. Specifically, we apply random masking exclusively to the student's global view, while preserving the student's local views and the teacher's global view in their original, unmasked forms. This design leverages DINO's multi-view augmentation scheme to retain clean supervision while inducing robustness through masked inputs. We evaluate our approach using DINO-Tiny on the mini-ImageNet dataset and show that random masking under this asymmetric setup yields more robust and fine-grained attention maps, ultimately enhancing downstream performance.