Enhancing Performance of Vision Transformers on Small Datasets through Local Inductive Bias Incorporation
This work addresses a specific bottleneck for researchers and practitioners using ViTs on limited data, though it is incremental as it builds on prior methods for adding locality to ViTs.
The paper tackles the problem of vision transformers (ViTs) underperforming on small datasets compared to CNNs by proposing a Local InFormation Enhancer (LIFE) module to incorporate local inductive bias, improving ViT performance on small image classification datasets and extending it to downstream tasks like object detection and semantic segmentation.
Vision transformers (ViTs) achieve remarkable performance on large datasets, but tend to perform worse than convolutional neural networks (CNNs) when trained from scratch on smaller datasets, possibly due to a lack of local inductive bias in the architecture. Recent studies have therefore added locality to the architecture and demonstrated that it can help ViTs achieve performance comparable to CNNs in the small-size dataset regime. Existing methods, however, are architecture-specific or have higher computational and memory costs. Thus, we propose a module called Local InFormation Enhancer (LIFE) that extracts patch-level local information and incorporates it into the embeddings used in the self-attention block of ViTs. Our proposed module is memory and computation efficient, as well as flexible enough to process auxiliary tokens such as the classification and distillation tokens. Empirical results show that the addition of the LIFE module improves the performance of ViTs on small image classification datasets. We further demonstrate how the effect can be extended to downstream tasks, such as object detection and semantic segmentation. In addition, we introduce a new visualization method, Dense Attention Roll-Out, specifically designed for dense prediction tasks, allowing the generation of class-specific attention maps utilizing the attention maps of all tokens.