Deeper Insights into the Robustness of ViTs towards Common Corruptions
This work addresses the robustness gap in ViTs for computer vision applications, offering incremental improvements through novel architectural and augmentation strategies.
The paper investigates the robustness of Vision Transformer (ViT) variants to common corruptions, identifying key architectural designs like overlapping patch embedding and convolutional feed-forward networks that enhance robustness, and introduces a dynamic augmentation method that achieves state-of-the-art results.
With Vision Transformers (ViTs) making great advances in a variety of computer vision tasks, recent literature have proposed various variants of vanilla ViTs to achieve better efficiency and efficacy. However, it remains unclear how their unique architecture impact robustness towards common corruptions. In this paper, we make the first attempt to probe into the robustness gap among ViT variants and explore underlying designs that are essential for robustness. Through an extensive and rigorous benchmarking, we demonstrate that simple architecture designs such as overlapping patch embedding and convolutional feed-forward network (FFN) can promote the robustness of ViTs. Moreover, since training ViTs relies heavily on data augmentation, whether previous CNN-based augmentation strategies that are targeted at robustness purposes can still be useful is worth investigating. We explore different data augmentation on ViTs and verify that adversarial noise training is powerful while fourier-domain augmentation is inferior. Based on these findings, we introduce a novel conditional method of generating dynamic augmentation parameters conditioned on input images, offering state-of-the-art robustness towards common corruptions.