AV-Unified: A Unified Framework for Audio-visual Scene Understanding
This work addresses the challenge of comprehensively understanding complex audio-visual scenes by enabling joint learning across multiple tasks, which is significant for researchers and developers working on multi-modal AI systems.
This paper proposes AV-Unified, a unified framework for joint learning across multiple audio-visual scene understanding tasks, including event localization, parsing, segmentation, and question answering. It standardizes diverse input-output formats into sequences of discrete tokens and incorporates a multi-scale spatiotemporal perception network to capture audio-visual associations, demonstrating effectiveness across various benchmark datasets.
When humans perceive the world, they naturally integrate multiple audio-visual tasks within dynamic, real-world scenes. However, current works such as event localization, parsing, segmentation and question answering are mostly explored individually, making it challenging to comprehensively understand complex audio-visual scenes and explore inter-task relationships. Hence, we propose \textbf{AV-Unified}, a unified framework that enables joint learning across a wide range of audio-visual scene understanding tasks. AV-Unified standardizes the diverse input-output formats of each task and incorporates a multi-scale spatiotemporal perception network to effectively capture audio-visual associations. Specifically, we unify the inputs and outputs of all supported tasks by converting them into sequences of discrete tokens, establishing a shared representation that allows a single architecture to be trained jointly across heterogeneous varied datasets. Considering the varying temporal granularity of audio-visual events, a multi-scale temporal perception module is designed to capture key cues. Meanwhile, to overcome the lack of auditory supervision in the visual domain, we design a cross-modal guidance-based spatial perception module that models spatial audio-visual associations. Furthermore, task-specific text prompts are employed to enhance the model's adaptability and task-awareness. Extensive experiments on benchmark datasets (e.g., AVE, LLP, MUSIC-AVQA, VGG-SS and AVS) demonstrate the effectiveness of AV-Unified across temporal, spatial, and spatiotemporal tasks.