AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation
This work addresses audio-visual tasks for applications in multimedia analysis, but it is incremental as it adapts an existing model to a new modality.
The paper tackles the problem of audio-visual localization and segmentation by proposing AV-SAM, a framework based on the Segment Anything Model, which generates sounding object masks corresponding to audio, achieving competitive performance on datasets like Flickr-SoundNet and AVSBench.
Segment Anything Model (SAM) has recently shown its powerful effectiveness in visual segmentation tasks. However, there is less exploration concerning how SAM works on audio-visual tasks, such as visual sound localization and segmentation. In this work, we propose a simple yet effective audio-visual localization and segmentation framework based on the Segment Anything Model, namely AV-SAM, that can generate sounding object masks corresponding to the audio. Specifically, our AV-SAM simply leverages pixel-wise audio-visual fusion across audio features and visual features from the pre-trained image encoder in SAM to aggregate cross-modal representations. Then, the aggregated cross-modal features are fed into the prompt encoder and mask decoder to generate the final audio-visual segmentation masks. We conduct extensive experiments on Flickr-SoundNet and AVSBench datasets. The results demonstrate that the proposed AV-SAM can achieve competitive performance on sounding object localization and segmentation.