Show and Segment: Universal Medical Image Segmentation via In-Context Learning
This addresses the problem of limited generalization in medical image segmentation for healthcare applications, offering a flexible approach that reduces the need for task-specific training, though it builds incrementally on in-context learning concepts.
The paper tackles the challenge of medical image segmentation across diverse anatomical structures and imaging modalities by introducing Iris, an in-context learning framework that adapts to novel tasks without fine-tuning, achieving strong performance on in-distribution tasks and superior generalization on out-of-distribution data across twelve datasets.
Medical image segmentation remains challenging due to the vast diversity of anatomical structures, imaging modalities, and segmentation tasks. While deep learning has made significant advances, current approaches struggle to generalize as they require task-specific training or fine-tuning on unseen classes. We present Iris, a novel In-context Reference Image guided Segmentation framework that enables flexible adaptation to novel tasks through the use of reference examples without fine-tuning. At its core, Iris features a lightweight context task encoding module that distills task-specific information from reference context image-label pairs. This rich context embedding information is used to guide the segmentation of target objects. By decoupling task encoding from inference, Iris supports diverse strategies from one-shot inference and context example ensemble to object-level context example retrieval and in-context tuning. Through comprehensive evaluation across twelve datasets, we demonstrate that Iris performs strongly compared to task-specific models on in-distribution tasks. On seven held-out datasets, Iris shows superior generalization to out-of-distribution data and unseen classes. Further, Iris's task encoding module can automatically discover anatomical relationships across datasets and modalities, offering insights into medical objects without explicit anatomical supervision.