Alleviating Textual Reliance in Medical Language-guided Segmentation via Prototype-driven Semantic Approximation
This work addresses a critical limitation for medical AI applications by enabling more flexible segmentation in clinical scenarios where textual reports are not always paired with images, though it is incremental in improving existing language-guided methods.
The paper tackles the problem of textual reliance in medical language-guided segmentation, where paired image-text data is often unavailable, by proposing ProLearn, a prototype-driven framework that approximates semantic guidance from text to enable training and inference on image-only data, achieving state-of-the-art performance on datasets like QaTa-COV19, MosMedData+, and Kvasir-SEG.
Medical language-guided segmentation, integrating textual clinical reports as auxiliary guidance to enhance image segmentation, has demonstrated significant improvements over unimodal approaches. However, its inherent reliance on paired image-text input, which we refer to as ``textual reliance", presents two fundamental limitations: 1) many medical segmentation datasets lack paired reports, leaving a substantial portion of image-only data underutilized for training; and 2) inference is limited to retrospective analysis of cases with paired reports, limiting its applicability in most clinical scenarios where segmentation typically precedes reporting. To address these limitations, we propose ProLearn, the first Prototype-driven Learning framework for language-guided segmentation that fundamentally alleviates textual reliance. At its core, we introduce a novel Prototype-driven Semantic Approximation (PSA) module to enable approximation of semantic guidance from textual input. PSA initializes a discrete and compact prototype space by distilling segmentation-relevant semantics from textual reports. Once initialized, it supports a query-and-respond mechanism which approximates semantic guidance for images without textual input, thereby alleviating textual reliance. Extensive experiments on QaTa-COV19, MosMedData+ and Kvasir-SEG demonstrate that ProLearn outperforms state-of-the-art language-guided methods when limited text is available.