NOVO: Bridging LLaVA and SAM with Visual-only Prompts for Reasoning Segmentation
This work addresses the challenge of enabling instance-level segmentation with reasoning for computer vision applications, representing an incremental improvement over prior methods.
The paper tackles the problem of integrating vision-language models with segmentation models for reasoning segmentation by proposing NOVO, which uses visual-only prompts to generate coarse masks and point prompts for SAM, achieving state-of-the-art performance across multiple metrics and model sizes.
In this study, we propose NOVO (NO text, Visual-Only prompts), a novel framework that bridges vision-language models (VLMs) and segmentation models through visual-only prompts. Unlike prior approaches that feed text-derived SEG token embeddings into segmentation models, NOVO instead generates a coarse mask and point prompts from the VLM output. These visual prompts are compatible with the Segment Anything Model (SAM), preserving alignment with its pretrained capabilities. To further enhance boundary quality and enable instance-level segmentation, we introduce a training-free refinement module that reduces visual artifacts and improves the quality of segmentation masks. We also present RISeg, a new benchmark comprising 918 images, 2,533 instance-level masks, and diverse reasoning queries to evaluate this task. Experiments demonstrate that NOVO achieves state-of-the-art performance across multiple metrics and model sizes, demonstrating its effectiveness and scalability in reasoning segmentation.