Towards Motion-aware Referring Image Segmentation
This work addresses a specific bottleneck in RIS for applications requiring action-based object identification, representing an incremental advancement.
The paper tackles the underperformance of Referring Image Segmentation (RIS) models on motion-related queries by introducing a data augmentation scheme for motion-centric phrases and a multimodal contrastive learning method, achieving substantial improvements on motion-centric queries while maintaining competitive results on appearance-based descriptions.
Referring Image Segmentation (RIS) requires identifying objects from images based on textual descriptions. We observe that existing methods significantly underperform on motion-related queries compared to appearance-based ones. To address this, we first introduce an efficient data augmentation scheme that extracts motion-centric phrases from original captions, exposing models to more motion expressions without additional annotations. Second, since the same object can be described differently depending on the context, we propose Multimodal Radial Contrastive Learning (MRaCL), performed on fused image-text embeddings rather than unimodal representations. For comprehensive evaluation, we introduce a new test split focusing on motion-centric queries, and introduce a new benchmark called M-Bench, where objects are distinguished primarily by actions. Extensive experiments show our method substantially improves performance on motion-centric queries across multiple RIS models, maintaining competitive results on appearance-based descriptions. Codes are available at https://github.com/snuviplab/MRaCL