Unsupervised Audio-Visual Segmentation with Modality Alignment
This addresses the scalability issue in audio-visual segmentation for researchers and practitioners by eliminating costly annotations, though it builds incrementally on existing foundation models.
The paper tackles the problem of audio-visual segmentation without expensive fine-grained annotations by proposing an unsupervised method called Modality Correspondence Alignment (MoCA), which integrates foundation models like DINO, SAM, and ImageBind to associate audio with visual objects at the pixel level, achieving substantial improvements in mIoU over baselines (e.g., +17.24% to +67.64% on AVSBench datasets).
Audio-Visual Segmentation (AVS) aims to identify, at the pixel level, the object in a visual scene that produces a given sound. Current AVS methods rely on costly fine-grained annotations of mask-audio pairs, making them impractical for scalability. To address this, we introduce unsupervised AVS, eliminating the need for such expensive annotation. To tackle this more challenging problem, we propose an unsupervised learning method, named Modality Correspondence Alignment (MoCA), which seamlessly integrates off-the-shelf foundation models like DINO, SAM, and ImageBind. This approach leverages their knowledge complementarity and optimizes their joint usage for multi-modality association. Initially, we estimate positive and negative image pairs in the feature space. For pixel-level association, we introduce an audio-visual adapter and a novel pixel matching aggregation strategy within the image-level contrastive learning framework. This allows for a flexible connection between object appearance and audio signal at the pixel level, with tolerance to imaging variations such as translation and rotation. Extensive experiments on the AVSBench (single and multi-object splits) and AVSS datasets demonstrate that our MoCA outperforms strongly designed baseline methods and approaches supervised counterparts, particularly in complex scenarios with multiple auditory objects. Notably when comparing mIoU, MoCA achieves a substantial improvement over baselines in both the AVSBench (S4: +17.24%; MS3: +67.64%) and AVSS (+19.23%) audio-visual segmentation challenges.