Toward Semantic-Agnostic and Shape-Aware Vision-Language Segmentation Models
For researchers in vision-language segmentation, this work proposes a new task and method to improve low- and mid-level visual reasoning, though it is incremental as it builds on existing models.
The paper introduces Semantic-Agnostic and Shape-Aware (SANSA) segmentation, a new paradigm requiring models to segment based on non-semantic textual descriptions. Finetuning on SANSA prompts yields up to 20% mIoU improvement on this task while maintaining performance on standard semantic prompts.
Vision-language segmentation models have recently achieved strong performance by leveraging high-level semantic object categories expressed in natural language. However, this semantic dependence limits their ability to reason about intrinsic visual properties such as shape, geometry, or texture, which are essential in many real-world applications. In this work, we introduce Semantic-Agnostic aNd Shape-Aware (SANSA) segmentation, a new paradigm that requires segmentation models to operate solely from non-semantic textual descriptions. To this end, we propose two strategies to generate SANSA segmentation prompts based on either dictionary constraints or example guidance, both generating semantic-agnostic textual descriptions. These prompts are then used to finetune segmentation models under semantic-agnostic supervision. Experiments show that finetuning on SANSA prompts yields up to a 20% mIoU improvement on this new segmentation task, compared to pretrained state-of-the-art models, while maintaining strong performance on standard semantic prompts. These results highlight the importance of low- and mid-level visual reasoning for improving the generalization and controllability of vision-language segmentation models.