SaSR-Net: Source-Aware Semantic Representation Network for Enhancing Audio-Visual Question Answering
This work addresses the problem of enhancing answer accuracy in AVQA for applications like video understanding, though it appears incremental as it builds on existing AVQA methods with novel model components.
The paper tackled the challenge of interpreting complex multi-modal scenes in Audio-Visual Question Answering (AVQA) by introducing SaSR-Net, which uses source-wise learnable tokens and attention mechanisms to align audio-visual elements with questions, achieving state-of-the-art performance on Music-AVQA and AVQA-Yang datasets.
Audio-Visual Question Answering (AVQA) is a challenging task that involves answering questions based on both auditory and visual information in videos. A significant challenge is interpreting complex multi-modal scenes, which include both visual objects and sound sources, and connecting them to the given question. In this paper, we introduce the Source-aware Semantic Representation Network (SaSR-Net), a novel model designed for AVQA. SaSR-Net utilizes source-wise learnable tokens to efficiently capture and align audio-visual elements with the corresponding question. It streamlines the fusion of audio and visual information using spatial and temporal attention mechanisms to identify answers in multi-modal scenes. Extensive experiments on the Music-AVQA and AVQA-Yang datasets show that SaSR-Net outperforms state-of-the-art AVQA methods.