Neuroverse3D: Developing In-Context Learning Universal Model for Neuroimaging in 3D
This work addresses the challenge of adapting universal models to 3D neuroimaging for medical centers, enabling flexible task adaptation without retraining, though it is incremental as it builds on existing ICL paradigms.
The paper tackles the problem of in-context learning (ICL) models being limited to 2D inputs and suboptimal for neuroimaging by introducing Neuroverse3D, an ICL model for 3D neuroimaging tasks like segmentation and denoising, which outperforms existing ICL models and closely matches task-specific models on 14 diverse tasks using 43,674 scans.
In-context learning (ICL), a type of universal model, demonstrates exceptional generalization across a wide range of tasks without retraining by leveraging task-specific guidance from context, making it particularly effective for the intricate demands of neuroimaging. However, current ICL models, limited to 2D inputs and thus exhibiting suboptimal performance, struggle to extend to 3D inputs due to the high memory demands of ICL. In this regard, we introduce Neuroverse3D, an ICL model capable of performing multiple neuroimaging tasks in 3D (e.g., segmentation, denoising, inpainting). Neuroverse3D overcomes the large memory consumption associated with 3D inputs through adaptive parallel-sequential context processing and a U-shaped fusion strategy, allowing it to handle an unlimited number of context images. Additionally, we propose an optimized loss function to balance multi-task training and enhance focus on anatomical boundaries. Our study incorporates 43,674 3D multi-modal scans from 19 neuroimaging datasets and evaluates Neuroverse3D on 14 diverse tasks using held-out test sets. The results demonstrate that Neuroverse3D significantly outperforms existing ICL models and closely matches task-specific models, enabling flexible adaptation to medical center variations without retraining. The code and model weights are publicly available at https://github.com/jiesihu/Neuroverse3D.