ViT-AE++: Improving Vision Transformer Autoencoder for Self-supervised Medical Image Representations
This work addresses the problem of learning effective representations from unlabeled medical images for researchers and practitioners, but it is incremental as it builds on an existing method.
The paper tackled improving self-supervised representation learning for medical images by enhancing the Vision Transformer Autoencoder (ViT-AE) with new loss functions, resulting in consistent improvements over vanilla ViT-AE and superiority over other contrastive learning approaches on both natural and medical images.
Self-supervised learning has attracted increasing attention as it learns data-driven representation from data without annotations. Vision transformer-based autoencoder (ViT-AE) by He et al. (2021) is a recent self-supervised learning technique that employs a patch-masking strategy to learn a meaningful latent space. In this paper, we focus on improving ViT-AE (nicknamed ViT-AE++) for a more effective representation of 2D and 3D medical images. We propose two new loss functions to enhance the representation during training. The first loss term aims to improve self-reconstruction by considering the structured dependencies and indirectly improving the representation. The second loss term leverages contrastive loss to optimize the representation from two randomly masked views directly. We extended ViT-AE++ to a 3D fashion for volumetric medical images as an independent contribution. We extensively evaluate ViT-AE++ on both natural images and medical images, demonstrating consistent improvement over vanilla ViT-AE and its superiority over other contrastive learning approaches. Codes are here: https://github.com/chinmay5/vit_ae_plus_plus.git.