SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenes
This work addresses the challenge of pixel-wise scene understanding in language-aided audio-visual scenes, which is an incremental advancement for applications in video analysis and multimodal AI.
The paper tackles the problem of segmenting objects in videos using both text and audio references, where previous methods struggled with spatio-temporal consistency, and achieves an 8.5% improvement in performance over the state-of-the-art on the Ref-AVS benchmark.
Reference Audio-Visual Segmentation (Ref-AVS) aims to provide a pixel-wise scene understanding in Language-aided Audio-Visual Scenes (LAVS). This task requires the model to continuously segment objects referred to by text and audio from a video. Previous dual-modality methods always fail due to the lack of a third modality and the existing triple-modality method struggles with spatio-temporal consistency, leading to the target shift of different frames. In this work, we introduce a novel framework, termed SAM2-LOVE, which integrates textual, audio, and visual representations into a learnable token to prompt and align SAM2 for achieving Ref-AVS in the LAVS. Technically, our approach includes a multimodal fusion module aimed at improving multimodal understanding of SAM2, as well as token propagation and accumulation strategies designed to enhance spatio-temporal consistency without forgetting historical information. We conducted extensive experiments to demonstrate that SAM2-LOVE outperforms the SOTA by 8.5\% in $\mathcal{J\&F}$ on the Ref-AVS benchmark and showcase the simplicity and effectiveness of the components. Our code will be available here.