AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answering
This work addresses the challenge of flexible and adaptive reasoning in audio-visual scenes for AI systems, representing an incremental advancement with strong domain-specific gains.
The paper tackled the problem of Audio-Visual Question Answering (AVQA) by proposing AV-Master, a framework that dynamically models temporal and modality dimensions to extract key information from complex scenes, resulting in significant performance improvements on four benchmarks, especially in complex reasoning tasks.
Audio-Visual Question Answering (AVQA) requires models to effectively utilize both visual and auditory modalities to answer complex and diverse questions about audio-visual scenes. However, existing methods lack sufficient flexibility and dynamic adaptability in temporal sampling and modality preference awareness, making it difficult to focus on key information based on the question. This limits their reasoning capability in complex scenarios. To address these challenges, we propose a novel framework named AV-Master. It enhances the model's ability to extract key information from complex audio-visual scenes with substantial redundant content by dynamically modeling both temporal and modality dimensions. In the temporal dimension, we introduce a dynamic adaptive focus sampling mechanism that progressively focuses on audio-visual segments most relevant to the question, effectively mitigating redundancy and segment fragmentation in traditional sampling methods. In the modality dimension, we propose a preference-aware strategy that models each modality's contribution independently, enabling selective activation of critical features. Furthermore, we introduce a dual-path contrastive loss to reinforce consistency and complementarity across temporal and modality dimensions, guiding the model to learn question-specific cross-modal collaborative representations. Experiments on four large-scale benchmarks show that AV-Master significantly outperforms existing methods, especially in complex reasoning tasks.