Pretrained ViTs Yield Versatile Representations For Medical Images
This provides a viable alternative to CNNs for medical imaging tasks, though it is incremental as it compares existing methods on new data.
The paper tackled the problem of evaluating vision transformers (ViTs) for medical image classification, finding that off-the-shelf ViTs pretrained on ImageNet perform on par with CNNs on standard 2D medical image benchmarks.
Convolutional Neural Networks (CNNs) have reigned for a decade as the de facto approach to automated medical image diagnosis, pushing the state-of-the-art in classification, detection and segmentation tasks. Over the last years, vision transformers (ViTs) have appeared as a competitive alternative to CNNs, yielding impressive levels of performance in the natural image domain, while possessing several interesting properties that could prove beneficial for medical imaging tasks. In this work, we explore the benefits and drawbacks of transformer-based models for medical image classification. We conduct a series of experiments on several standard 2D medical image benchmark datasets and tasks. Our findings show that, while CNNs perform better if trained from scratch, off-the-shelf vision transformers can perform on par with CNNs when pretrained on ImageNet, both in a supervised and self-supervised setting, rendering them as a viable alternative to CNNs.