Medical Transformer: Universal Brain Encoder for 3D MRI Analysis
This work addresses the challenge of high computational costs for medical practitioners using deep learning on limited annotated 3D MRI datasets, offering a more efficient solution.
The authors tackled the problem of parameter inefficiency in 3D medical image analysis by proposing Medical Transformer, a transfer learning framework that models 3D MRI volumes as sequences of 2D slices with multi-view processing, achieving up to 92% parameter reduction while outperforming state-of-the-art methods on tasks like brain disease diagnosis, age prediction, and tumor segmentation.
Transfer learning has gained attention in medical image analysis due to limited annotated 3D medical datasets for training data-driven deep learning models in the real world. Existing 3D-based methods have transferred the pre-trained models to downstream tasks, which achieved promising results with only a small number of training samples. However, they demand a massive amount of parameters to train the model for 3D medical imaging. In this work, we propose a novel transfer learning framework, called Medical Transformer, that effectively models 3D volumetric images in the form of a sequence of 2D image slices. To make a high-level representation in 3D-form empowering spatial relations better, we take a multi-view approach that leverages plenty of information from the three planes of 3D volume, while providing parameter-efficient training. For building a source model generally applicable to various tasks, we pre-train the model in a self-supervised learning manner for masked encoding vector prediction as a proxy task, using a large-scale normal, healthy brain magnetic resonance imaging (MRI) dataset. Our pre-trained model is evaluated on three downstream tasks: (i) brain disease diagnosis, (ii) brain age prediction, and (iii) brain tumor segmentation, which are actively studied in brain MRI research. The experimental results show that our Medical Transformer outperforms the state-of-the-art transfer learning methods, efficiently reducing the number of parameters up to about 92% for classification and