A Benchmark for Audio Reasoning Capabilities of Multimodal Large Language Models
This work addresses a gap in evaluating audio reasoning for multimodal models, but it is incremental as it builds on existing benchmarks by adding a new testing framework.
The authors tackled the lack of benchmarks for evaluating multimodal large language models' ability to reason across different audio tasks, and they proposed a new benchmark called Audio Reasoning Tasks (ART) to assess this capability.
The present benchmarks for testing the audio modality of multimodal large language models concentrate on testing various audio tasks such as speaker diarization or gender identification in isolation. Whether a multimodal model can answer the questions that require reasoning skills to combine audio tasks of different categories, cannot be verified with their use. To address this issue, we propose Audio Reasoning Tasks (ART), a new benchmark for assessing the ability of multimodal models to solve problems that require reasoning over audio signal.