Training data-efficient image transformers & distillation through attention
This work makes vision transformers more accessible to researchers and practitioners without access to massive datasets and expensive infrastructure, by significantly reducing training data and computational requirements.
This paper addresses the high data and computational cost of training vision transformers by training a competitive convolution-free transformer on ImageNet only, achieving 83.1% top-1 accuracy. They also introduce a teacher-student distillation strategy using a distillation token, which boosts accuracy to 85.2% on ImageNet.
Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. However, these visual transformers are pre-trained with hundreds of millions of images using an expensive infrastructure, thereby limiting their adoption. In this work, we produce a competitive convolution-free transformer by training on Imagenet only. We train them on a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop evaluation) on ImageNet with no external data. More importantly, we introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention. We show the interest of this token-based distillation, especially when using a convnet as a teacher. This leads us to report results competitive with convnets for both Imagenet (where we obtain up to 85.2% accuracy) and when transferring to other tasks. We share our code and models.