SeLIP: Similarity Enhanced Contrastive Language Image Pretraining for Multi-modal Head MRI
This addresses the data scarcity issue for medical image analysis, though it is incremental as it builds on existing contrastive learning methods.
The paper tackled the problem of limited annotated medical data for deep learning by proposing SeLIP, a foundation model for multi-modal head MRI that uses contrastive learning with a mixed similarity metric, resulting in effective performance in downstream tasks like retrieval, classification, and segmentation.
Despite that deep learning (DL) methods have presented tremendous potential in many medical image analysis tasks, the practical applications of medical DL models are limited due to the lack of enough data samples with manual annotations. By noting that the clinical radiology examinations are associated with radiology reports that describe the images, we propose to develop a foundation model for multi-model head MRI by using contrastive learning on the images and the corresponding radiology findings. In particular, a contrastive learning framework is proposed, where a mixed syntax and semantic similarity matching metric is integrated to reduce the thirst of extreme large dataset in conventional contrastive learning framework. Our proposed similarity enhanced contrastive language image pretraining (SeLIP) is able to effectively extract more useful features. Experiments revealed that our proposed SeLIP performs well in many downstream tasks including image-text retrieval task, classification task, and image segmentation, which highlights the importance of considering the similarities among texts describing different images in developing medical image foundation models.