Multi-modal Vision Pre-training for Medical Image Analysis
This work addresses the need for effective cross-modal representation learning in medical imaging, particularly for multi-modal data like MRI scans, offering incremental improvements over existing methods.
The paper tackled the problem of neglecting inter-modal correlations in self-supervised learning for medical image analysis by introducing a novel multi-modal pre-training method with three proxy tasks, achieving Dice Score improvements of 0.28%-14.47% in segmentation and accuracy boosts of 0.65%-18.07% in classification tasks.
Self-supervised learning has greatly facilitated medical image analysis by suppressing the training data requirement for real-world applications. Current paradigms predominantly rely on self-supervision within uni-modal image data, thereby neglecting the inter-modal correlations essential for effective learning of cross-modal image representations. This limitation is particularly significant for naturally grouped multi-modal data, e.g., multi-parametric MRI scans for a patient undergoing various functional imaging protocols in the same study. To bridge this gap, we conduct a novel multi-modal image pre-training with three proxy tasks to facilitate the learning of cross-modality representations and correlations using multi-modal brain MRI scans (over 2.4 million images in 16,022 scans of 3,755 patients), i.e., cross-modal image reconstruction, modality-aware contrastive learning, and modality template distillation. To demonstrate the generalizability of our pre-trained model, we conduct extensive experiments on various benchmarks with ten downstream tasks. The superior performance of our method is reported in comparison to state-of-the-art pre-training methods, with Dice Score improvement of 0.28\%-14.47\% across six segmentation benchmarks and a consistent accuracy boost of 0.65\%-18.07\% in four individual image classification tasks.