CORA: Consistency-Guided Semi-Supervised Framework for Reasoning Segmentation
This addresses the high cost of annotation for reasoning segmentation in domains like urban scenes and histopathology, offering a more efficient solution, though it is incremental as it builds on existing multimodal language models.
The paper tackles the problem of reasoning segmentation, which requires pixel-accurate masks from complex instructions, by proposing CORA, a semi-supervised framework that uses limited labeled data and unlabeled images to improve generalization. It achieves state-of-the-art results, such as a +2.3% improvement with only 100 labeled images on Cityscapes and +2.4% with 180 labeled images on PanNuke.
Reasoning segmentation seeks pixel-accurate masks for targets referenced by complex, often implicit instructions, requiring context-dependent reasoning over the scene. Recent multimodal language models have advanced instruction following segmentation, yet generalization remains limited. The key bottleneck is the high cost of curating diverse, high-quality pixel annotations paired with rich linguistic supervision leading to brittle performance under distribution shift. Therefore, we present CORA, a semi-supervised reasoning segmentation framework that jointly learns from limited labeled data and a large corpus of unlabeled images. CORA introduces three main components: 1) conditional visual instructions that encode spatial and contextual relationships between objects; 2) a noisy pseudo-label filter based on the consistency of Multimodal LLM's outputs across semantically equivalent queries; and 3) a token-level contrastive alignment between labeled and pseudo-labeled samples to enhance feature consistency. These components enable CORA to perform robust reasoning segmentation with minimal supervision, outperforming existing baselines under constrained annotation settings. CORA achieves state-of-the-art results, requiring as few as 100 labeled images on Cityscapes, a benchmark dataset for urban scene understanding, surpassing the baseline by $+2.3\%$. Similarly, CORA improves performance by $+2.4\%$ with only 180 labeled images on PanNuke, a histopathology dataset.