SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentation
It addresses a key limitation in referring image segmentation for real-world applications by handling ambiguous expressions, though it is incremental as it builds on existing methods with a novel framework.
The paper tackles the challenge of referring image segmentation with ambiguous natural language expressions, such as object-distracting and category-implicit cases, by proposing SaFiRe, a framework that mimics human cognitive processes using Mamba, and achieves superior performance over state-of-the-art baselines on standard and new benchmarks.
Referring Image Segmentation (RIS) aims to segment the target object in an image given a natural language expression. While recent methods leverage pre-trained vision backbones and more training corpus to achieve impressive results, they predominantly focus on simple expressions--short, clear noun phrases like "red car" or "left girl". This simplification often reduces RIS to a key word/concept matching problem, limiting the model's ability to handle referential ambiguity in expressions. In this work, we identify two challenging real-world scenarios: object-distracting expressions, which involve multiple entities with contextual cues, and category-implicit expressions, where the object class is not explicitly stated. To address the challenges, we propose a novel framework, SaFiRe, which mimics the human two-phase cognitive process--first forming a global understanding, then refining it through detail-oriented inspection. This is naturally supported by Mamba's scan-then-update property, which aligns with our phased design and enables efficient multi-cycle refinement with linear complexity. We further introduce aRefCOCO, a new benchmark designed to evaluate RIS models under ambiguous referring expressions. Extensive experiments on both standard and proposed datasets demonstrate the superiority of SaFiRe over state-of-the-art baselines.