Efficient In-Context Medical Segmentation with Meta-driven Visual Prompt Selection
This addresses the challenge of domain shifts and prompt dependency in medical segmentation for researchers and practitioners, offering a flexible, data-centric solution that is incremental over existing model-centric approaches.
The paper tackles the problem of in-context learning for medical image segmentation by proposing a meta-driven method to select optimal visual prompts, improving performance and generalizability across 8 datasets and 4 tasks with gains in computational and label efficiency.
In-context learning (ICL) with Large Vision Models (LVMs) presents a promising avenue in medical image segmentation by reducing the reliance on extensive labeling. However, the ICL performance of LVMs highly depends on the choices of visual prompts and suffers from domain shifts. While existing works leveraging LVMs for medical tasks have focused mainly on model-centric approaches like fine-tuning, we study an orthogonal data-centric perspective on how to select good visual prompts to facilitate generalization to medical domain. In this work, we propose a label-efficient in-context medical segmentation method by introducing a novel Meta-driven Visual Prompt Selection mechanism (MVPS), where a prompt retriever obtained from a meta-learning framework actively selects the optimal images as prompts to promote model performance and generalizability. Evaluated on 8 datasets and 4 tasks across 3 medical imaging modalities, our proposed approach demonstrates consistent gains over existing methods under different scenarios, improving both computational and label efficiency. Finally, we show that MVPS is a flexible, finetuning-free module that could be easily plugged into different backbones and combined with other model-centric approaches.