Hear to See: Discerning Stateful Listening for Audio-Visual Instance Segmentation
This work provides a strong improvement for the domain-specific task of audio-visual instance segmentation, offering a new SOTA for researchers in that niche.
The paper tackles audio-visual instance segmentation (AVIS), specifically the challenges of matching overlapping sound sources to visual instances and handling asynchronous audio-visual dynamics. The proposed Hear to See (H2S) method achieves state-of-the-art performance on AVISeg with 48.54 mAP, surpassing the previous best by 7.8%.
Audio-visual instance segmentation (AVIS) requires accurately identifying and tracking individual sounding objects with pixel-level masks. Existing methods struggle to match overlapping acoustic events with visual instances and handle asynchronous audio-visual dynamics. Therefore, two critical questions arise: how can a model establish precise correspondence between overlapping sound sources and visual instances, and how can a model maintain robust tracking when audio and visual signals are temporally misaligned?This paper proposes Hear to See (H2S), addressing these challenges through two mechanisms. The Acoustic-Semantic Projector (ASP) disentangles mixed audio and establishes hierarchical correspondence from semantic to spatial domains. The Asynchronous Dynamics Modulator (ADM) adaptively adjusts state transitions via audio-modulated Mamba, prioritizing current information during dynamic variations and maintaining continuity in stable periods.Experiments on AVISeg show H2S achieves SOTA performance, attaining 48.54 mAP with a COCO pretrained ResNet50 and surpassing the previous by 7.8\%. The code will be open-sourced once the paper is accepted. The source code will be publicly available at https://github.com/leiyeliu/H2S.