Structured Initialization for Attention in Vision Transformers
This addresses the problem of data-efficient ViT learning for researchers and practitioners, offering a method that is incremental but effective in bridging the gap with CNNs on small datasets.
The paper tackles the challenge of training vision transformers (ViTs) on small-scale datasets by reinterpreting the architectural bias of convolutional neural networks (CNNs) as an initialization bias for ViTs, achieving state-of-the-art performance on benchmarks like CIFAR-10, CIFAR-100, and SVHN.
The training of vision transformer (ViT) networks on small-scale datasets poses a significant challenge. By contrast, convolutional neural networks (CNNs) have an architectural inductive bias enabling them to perform well on such problems. In this paper, we argue that the architectural bias inherent to CNNs can be reinterpreted as an initialization bias within ViT. This insight is significant as it empowers ViTs to perform equally well on small-scale problems while maintaining their flexibility for large-scale applications. Our inspiration for this ``structured'' initialization stems from our empirical observation that random impulse filters can achieve comparable performance to learned filters within CNNs. Our approach achieves state-of-the-art performance for data-efficient ViT learning across numerous benchmarks including CIFAR-10, CIFAR-100, and SVHN.