Sub-Semantic Image Segmentation
This work proposes a new segmentation paradigm for researchers interested in fine-grained, language-guided image partitioning, though the contribution is incremental as it builds on existing models.
The paper introduces sub-semantic image segmentation, a new task that partitions images into stable appearance patterns described by language rather than naming whole objects. The proposed method, DETECTURE, addresses three failure modes in coupling vision-language models with SAM 3 and achieves state-of-the-art performance on several datasets.
Images can be segmented based on visual cues (i.e., texture segmentation) or into objects (i.e., semantic segmentation). We propose a new category of sub-semantic image segmentation that blurs the line between the two. In sub-semantic image segmentation, language is not used to name whole objects. Instead, it is used to partition an image into stable appearance patterns that can be described by language. To do that, we couple a general-purpose vision-language model to SAM 3, a promptable segmentation backbone whose native text pathway can ground rich descriptions into masks. Simple coupling fails for a number of reasons that we identify in the paper, and we overcome them by introducing DETECTURE that resolves three concrete failure modes -- language leakage between texture regions, prompt competition inside the segmentation backbone, and semantic distortion at the language-to-mask interface. Since there is no dataset of sub-semantic image segmentation, we introduce one, termed TextureADE. The new dataset is derived from the ADE20K dataset using a system we designed. We compare DETECTURE to a number of baselines and find that it achieves the strongest performance on several datasets using different metrics. Code is available at https://github.com/Scientific-Computing-Lab/TextureDetecture.