Discrete Representations Strengthen Vision Transformer Robustness
This addresses robustness issues in ViTs for image recognition, offering a simple modification to enhance generalization, though it is incremental as it builds on existing ViT architectures.
The paper tackles the problem that Vision Transformers (ViTs) overly rely on local textures and fail to use shape information, leading to poor generalization on out-of-distribution data, by adding discrete tokens from a vector-quantized encoder to ViT's input layer, which improves robustness by up to 12% across seven benchmarks while maintaining ImageNet performance.
Vision Transformer (ViT) is emerging as the state-of-the-art architecture for image recognition. While recent studies suggest that ViTs are more robust than their convolutional counterparts, our experiments find that ViTs trained on ImageNet are overly reliant on local textures and fail to make adequate use of shape information. ViTs thus have difficulties generalizing to out-of-distribution, real-world data. To address this deficiency, we present a simple and effective architecture modification to ViT's input layer by adding discrete tokens produced by a vector-quantized encoder. Different from the standard continuous pixel tokens, discrete tokens are invariant under small perturbations and contain less information individually, which promote ViTs to learn global information that is invariant. Experimental results demonstrate that adding discrete representation on four architecture variants strengthens ViT robustness by up to 12% across seven ImageNet robustness benchmarks while maintaining the performance on ImageNet.