Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos
This addresses the need for comprehensive multi-modal AI that can handle both static and dynamic visual content in complex real-world applications, though it appears incremental as it builds on existing models like SAM-2 and LLaVA.
The authors tackled the problem of dense grounded understanding of both images and videos by developing Sa2VA, a unified model that combines SAM-2 and LLaVA to support tasks like referring segmentation and conversation, achieving strong performance in referring video object segmentation with minimal one-shot tuning.
This work presents Sa2VA, the first comprehensive, unified model for dense grounded understanding of both images and videos. Unlike existing multi-modal large language models, which are often limited to specific modalities and tasks, Sa2VA supports a wide range of image and video tasks, including referring segmentation and conversation, with minimal one-shot instruction tuning. Sa2VA combines SAM-2, a foundation video segmentation model, with MLLM, the advanced vision-language model, and unifies text, image, and video into a shared LLM token space. Using the LLM, Sa2VA generates instruction tokens that guide SAM-2 in producing precise masks, enabling a grounded, multi-modal understanding of both static and dynamic visual content. Additionally, we introduce Ref-SAV, an auto-labeled dataset containing over 72k object expressions in complex video scenes, designed to boost model performance. We also manually validate 2k video objects in the Ref-SAV datasets to benchmark referring video object segmentation in complex environments. Experiments show that Sa2VA achieves strong performance across multiple tasks, particularly in referring video object segmentation, highlighting its potential for complex real-world applications. In addition, Sa2VA can be easily extended into various VLMs, including Qwen-VL and Intern-VL, which can be updated with rapid process in current open-sourced VLMs. Code and models have been provided to the community.