MDAR: A Multi-scene Dynamic Audio Reasoning Benchmark
This provides a new benchmark for evaluating AI models on dynamic audio reasoning, addressing a gap in existing static or single-scene benchmarks, though it is incremental as it builds on prior audio-language model research.
The authors tackled the lack of benchmarks for complex, multi-scene audio reasoning by introducing MDAR, a dataset with 3,000 question-answer pairs across diverse audio clips, and found that state-of-the-art models like Qwen2.5-Omni and GPT-4o Audio achieve up to 76.67% accuracy but none reach 80% across all tasks.
The ability to reason from audio, including speech, paralinguistic cues, environmental sounds, and music, is essential for AI agents to interact effectively in real-world scenarios. Existing benchmarks mainly focus on static or single-scene settings and do not fully capture scenarios where multiple speakers, unfolding events, and heterogeneous audio sources interact. To address these challenges, we introduce MDAR, a benchmark for evaluating models on complex, multi-scene, and dynamically evolving audio reasoning tasks. MDAR comprises 3,000 carefully curated question-answer pairs linked to diverse audio clips, covering five categories of complex reasoning and spanning three question types. We benchmark 26 state-of-the-art audio language models on MDAR and observe that they exhibit limitations in complex reasoning tasks. On single-choice questions, Qwen2.5-Omni (open-source) achieves 76.67% accuracy, whereas GPT-4o Audio (closed-source) reaches 68.47%; however, GPT-4o Audio substantially outperforms Qwen2.5-Omni on the more challenging multiple-choice and open-ended tasks. Across all three question types, no model achieves 80% performance. These findings underscore the unique challenges posed by MDAR and its value as a benchmark for advancing audio reasoning research.Code and benchmark can be found at https://github.com/luckyerr/MDAR.