Powerful Design of Small Vision Transformer on CIFAR10
This work addresses the problem of inefficient ViT designs for small datasets, offering incremental improvements for researchers and practitioners in computer vision.
The paper tackled the underperformance of Vision Transformers on small datasets by optimizing Tiny ViTs on CIFAR-10, finding that low-rank compression reduces redundancy with minimal loss and multiple CLS tokens boost accuracy.
Vision Transformers (ViTs) have demonstrated remarkable success on large-scale datasets, but their performance on smaller datasets often falls short of convolutional neural networks (CNNs). This paper explores the design and optimization of Tiny ViTs for small datasets, using CIFAR-10 as a benchmark. We systematically evaluate the impact of data augmentation, patch token initialization, low-rank compression, and multi-class token strategies on model performance. Our experiments reveal that low-rank compression of queries in Multi-Head Latent Attention (MLA) incurs minimal performance loss, indicating redundancy in ViTs. Additionally, introducing multiple CLS tokens improves global representation capacity, boosting accuracy. These findings provide a comprehensive framework for optimizing Tiny ViTs, offering practical insights for efficient and effective designs. Code is available at https://github.com/erow/PoorViTs.