AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Prompting
This work addresses the underexplored audio-visual fusion in segmentation models for applications like video analysis, though it appears incremental as it builds on SAM2 with specific enhancements.
The paper tackles the problem of integrating audio with the Segment Anything Model 2 (SAM2) for promptable segmentation in videos, proposing AuralSAM2 with an AuralFuser module and audio-guided contrastive learning, which achieves remarkable improvements over previous methods on public benchmarks.
Segment Anything Model 2 (SAM2) exhibits strong generalisation for promptable segmentation in video clips; however, its integration with the audio modality remains underexplored. Existing approaches mainly follow two directions: (1) injecting adapters into the image encoder to receive audio signals, which incurs efficiency costs during prompt engineering, and (2) leveraging additional foundation models to generate visual prompts for the sounding objects, which are often imprecisely localised, leading to misguidance in SAM2. Moreover, these methods overlook the rich semantic interplay between hierarchical visual features and other modalities, resulting in suboptimal cross-modal fusion. In this work, we propose AuralSAM2, comprising the novel AuralFuser module, which externally attaches to SAM2 to integrate features from different modalities and generate feature-level prompts, guiding SAM2's decoder in segmenting sounding targets. Such integration is facilitated by a feature pyramid, further refining semantic understanding and enhancing object awareness in multimodal scenarios. Additionally, the audio-guided contrastive learning is introduced to explicitly align audio and visual representations and to also mitigate biases caused by dominant visual patterns. Results on public benchmarks show that our approach achieves remarkable improvements over the previous methods in the field. Code is available at https://github.com/yyliu01/AuralSAM2.