Token Boosting for Robust Self-Supervised Visual Transformer Pre-training
This addresses the challenge of unreliable data in self-supervised learning for computer vision, offering an incremental improvement with practical benefits.
The paper tackles the problem of pre-training Visual Transformers on corrupted data by introducing the Token Boosting Module, which improves robustness and leads to consistent performance gains on downstream tasks across four corrupted datasets.
Learning with large-scale unlabeled data has become a powerful tool for pre-training Visual Transformers (VTs). However, prior works tend to overlook that, in real-world scenarios, the input data may be corrupted and unreliable. Pre-training VTs on such corrupted data can be challenging, especially when we pre-train via the masked autoencoding approach, where both the inputs and masked ``ground truth" targets can potentially be unreliable in this case. To address this limitation, we introduce the Token Boosting Module (TBM) as a plug-and-play component for VTs that effectively allows the VT to learn to extract clean and robust features during masked autoencoding pre-training. We provide theoretical analysis to show how TBM improves model pre-training with more robust and generalizable representations, thus benefiting downstream tasks. We conduct extensive experiments to analyze TBM's effectiveness, and results on four corrupted datasets demonstrate that TBM consistently improves performance on downstream tasks.