Audio-centric Video Understanding Benchmark without Text Shortcut
This work addresses the need for better audio-focused benchmarks in video understanding for researchers and developers of multimodal AI systems, though it is incremental as it builds on existing benchmark frameworks.
The authors tackled the problem of evaluating multimodal LLMs' video understanding by focusing on auditory information, proposing the AVUT benchmark with audio-centric tasks and a filtering mechanism to avoid text shortcuts, and found deficiencies in current models through evaluation.
Audio often serves as an auxiliary modality in video understanding tasks of audio-visual large language models (LLMs), merely assisting in the comprehension of visual information. However, a thorough understanding of videos significantly depends on auditory information, as audio offers critical context, emotional cues, and semantic meaning that visual data alone often lacks. This paper proposes an audio-centric video understanding benchmark (AVUT) to evaluate the video comprehension capabilities of multimodal LLMs with a particular focus on auditory information. AVUT introduces a suite of carefully designed audio-centric tasks, holistically testing the understanding of both audio content and audio-visual interactions in videos. Moreover, this work points out the text shortcut problem that largely exists in other benchmarks where the correct answer can be found from question text alone without needing videos. AVUT addresses this problem by proposing a answer permutation-based filtering mechanism. A thorough evaluation across a diverse range of open-source and proprietary multimodal LLMs is performed, followed by the analyses of deficiencies in audio-visual LLMs. Demos and data are available at https://github.com/lark-png/AVUT.