A Foundation Model for Brain MRI with Dynamic Modality Integration
This addresses the need for flexible and adaptable models in medical imaging, particularly for brain MRI analysis, but appears incremental as it builds on existing foundation model and self-supervised learning approaches.
The paper tackles the problem of handling different combinations of imaging sequences in brain MRI by developing a foundation model that uses a single encoder with learnable modality embeddings and a masked autoencoding objective, trained on about 60,000 multi-center MRIs, with preliminary results showing feasibility.
We present a foundation model for brain MRI that can work with different combinations of imaging sequences. The model uses one encoder with learnable modality embeddings, conditional layer normalization, and a masked autoencoding objective that accounts for missing modalities. A variance-covariance regularizer is applied to stabilize feature learning and improve representation diversity. This design removes the need for separate models for each modality and allows the network to adapt when some sequences are missing or unseen. It is trained on about 60,000 multi-center MRIs using self-supervised reconstruction and modality imputation to learn flexible representations. A learnable modality embedding guides feature extraction so the encoder can adjust to different inputs. We describe our planned evaluation on brain tumor and multiple sclerosis segmentation, as well as lesion classification, under various modality settings. Preliminary results show that the method works feasibly, and further experiments are planned to study its performance in more detail. All code and pretrained models are available at https://github.com/BrainFM/brainfm