What's Making That Sound Right Now? Video-centric Audio-Visual Localization
This work addresses the problem of accurately localizing sound sources in videos for applications in multimedia analysis, offering a more comprehensive benchmark but is incremental in advancing temporal modeling within the domain.
The paper tackled the problem of audio-visual localization by addressing limitations in existing methods that ignore temporal dynamics and assume simplified scenarios, proposing a new benchmark (AVATAR) and model (TAVLO) that achieve robust and precise alignment through high-resolution temporal modeling.
Audio-Visual Localization (AVL) aims to identify sound-emitting sources within a visual scene. However, existing studies focus on image-level audio-visual associations, failing to capture temporal dynamics. Moreover, they assume simplified scenarios where sound sources are always visible and involve only a single object. To address these limitations, we propose AVATAR, a video-centric AVL benchmark that incorporates high-resolution temporal information. AVATAR introduces four distinct scenarios -- Single-sound, Mixed-sound, Multi-entity, and Off-screen -- enabling a more comprehensive evaluation of AVL models. Additionally, we present TAVLO, a novel video-centric AVL model that explicitly integrates temporal information. Experimental results show that conventional methods struggle to track temporal variations due to their reliance on global audio features and frame-level mappings. In contrast, TAVLO achieves robust and precise audio-visual alignment by leveraging high-resolution temporal modeling. Our work empirically demonstrates the importance of temporal dynamics in AVL and establishes a new standard for video-centric audio-visual localization.