VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos
This addresses the challenge of precise video-text alignment for applications requiring detailed spatial and temporal understanding, representing a strong specific gain rather than a foundational advancement.
The paper tackles the problem of fine-grained pixel-level visual grounding in videos by introducing VideoGLaMM, a large multimodal model that integrates a large language model, dual vision encoder, and spatio-temporal decoder, achieving consistent outperformance over existing approaches on tasks like Grounded Conversation Generation, Visual Grounding, and Referring Video Segmentation.
Fine-grained alignment between videos and text is challenging due to complex spatial and temporal dynamics in videos. Existing video-based Large Multimodal Models (LMMs) handle basic conversations but struggle with precise pixel-level grounding in videos. To address this, we introduce VideoGLaMM, a LMM designed for fine-grained pixel-level grounding in videos based on user-provided textual inputs. Our design seamlessly connects three key components: a Large Language Model, a dual vision encoder that emphasizes both spatial and temporal details, and a spatio-temporal decoder for accurate mask generation. This connection is facilitated via tunable V-L and L-V adapters that enable close Vision-Language (VL) alignment. The architecture is trained to synchronize both spatial and temporal elements of video content with textual instructions. To enable fine-grained grounding, we curate a multimodal dataset featuring detailed visually-grounded conversations using a semiautomatic annotation pipeline, resulting in a diverse set of 38k video-QA triplets along with 83k objects and 671k masks. We evaluate VideoGLaMM on three challenging tasks: Grounded Conversation Generation, Visual Grounding, and Referring Video Segmentation. Experimental results show that our model consistently outperforms existing approaches across all three tasks.