JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation
This addresses the problem of limited evaluation tools for multi-modal AI models, particularly for researchers and developers working on audio-visual understanding, though it is incremental as it builds on existing benchmark concepts.
The authors tackled the lack of a comprehensive benchmark for evaluating joint audio-visual reasoning in Omni-LLMs by introducing JointAVBench, which includes multi-modal dependencies, diverse audio types, and varying scene spans, and found that the best-performing Omni-LLM achieved only 62.6% accuracy, highlighting significant gaps in performance.
Understanding videos inherently requires reasoning over both visual and auditory information. To properly evaluate Omni-Large Language Models (Omni-LLMs), which are capable of processing multi-modal information including vision and audio, an effective benchmark must comprehensively cover three key aspects: (1) multi-modal dependency (i.e., questions that cannot be answered using vision or audio alone), (2) diverse audio information types (e.g., speech, sound events), and (3) varying scene spans. However, existing datasets fall short in one or more of these dimensions, limiting strict and comprehensive evaluation. To address this gap, we introduce JointAVBench, a novel benchmark with strict audio-video correlation, spanning five cognitive dimensions, four audio information types (speech, sound events, music, vocal traits), and three scene spans (single-, cross-, and full-scene). Given the high cost of manual annotation, we propose an automated pipeline that leverages state-of-the-art vision-LLMs, audio-LLMs, and general-purpose LLMs to synthesize questions and answers that strictly require joint audio-visual understanding. We evaluate leading vision-only, audio-only, and Omni-LLMs on our dataset. Results show that even the best-performing Omni-LLM achieves an average accuracy of only 62.6\%, outperforming uni-modal baselines but revealing substantial room for improvement, especially in cross-scene reasoning.