RaViTT: Random Vision Transformer Tokens
This addresses the challenge of applying ViTs in data-scarce domains like biomedical imaging, though it is an incremental improvement over existing augmentation methods.
The paper tackles the problem of limited annotated data in Vision Transformers (ViTs) by proposing RaViTT, a random patch sampling strategy, which increases baseline accuracy across four datasets and outperforms state-of-the-art augmentation techniques in three datasets by margins of +1.23% to +4.32%.
Vision Transformers (ViTs) have successfully been applied to image classification problems where large annotated datasets are available. On the other hand, when fewer annotations are available, such as in biomedical applications, image augmentation techniques like introducing image variations or combinations have been proposed. However, regarding ViT patch sampling, less has been explored outside grid-based strategies. In this work, we propose Random Vision Transformer Tokens (RaViTT), a random patch sampling strategy that can be incorporated into existing ViTs. We experimentally evaluated RaViTT for image classification, comparing it with a baseline ViT and state-of-the-art (SOTA) augmentation techniques in 4 datasets, including ImageNet-1k and CIFAR-100. Results show that RaViTT increases the accuracy of the baseline in all datasets and outperforms the SOTA augmentation techniques in 3 out of 4 datasets by a significant margin +1.23% to +4.32%. Interestingly, RaViTT accuracy improvements can be achieved even with fewer tokens, thus reducing the computational load of any ViT model for a given accuracy value.