Not Quite Anything: Overcoming SAMs Limitations for 3D Medical Imaging
This work addresses domain shift in medical imaging for researchers and clinicians, offering a fast and label-efficient method without retraining foundation models, though it is incremental as it builds on existing models with a novel integration.
The paper tackled the problem of foundation segmentation models like SAM and SAM-2 struggling with 3D medical images such as brain MRIs due to low contrast and lack of sharp boundaries, by proposing a compositional approach that treats the foundation model output as an additional input channel, achieving about 96% volume accuracy on basal ganglia segmentation.
Foundation segmentation models such as SAM and SAM-2 perform well on natural images but struggle with brain MRIs where structures like the caudate and thalamus lack sharp boundaries and have low contrast. Rather than fine tune these models (for example MedSAM), we propose a compositional alternative where the foundation model output is treated as an additional input channel and passed alongside the MRI to highlight regions of interest. We generate SAM-2 prompts by using a lightweight 3D U-Net that was previously trained on MRI segmentation. The U-Net may have been trained on a different dataset, so its guesses are often imprecise but usually in the correct region. The edges of the resulting foundation model guesses are smoothed to improve alignment with the MRI. We also test prompt free segmentation using DINO attention maps in the same framework. This has-a architecture avoids modifying foundation weights and adapts to domain shift without retraining the foundation model. It reaches about 96 percent volume accuracy on basal ganglia segmentation, which is sufficient for our study of longitudinal volume change. The approach is fast, label efficient, and robust to out of distribution scans. We apply it to study inflammation linked changes in sudden onset pediatric OCD.