AudioBench: A Universal Benchmark for Audio Large Language Models
This addresses the lack of a comprehensive benchmark for AudioLLMs, providing an open-sourced toolkit for researchers in audio AI, though it is incremental as it builds on existing evaluation frameworks.
The authors introduced AudioBench, a universal benchmark with 8 tasks and 26 datasets (including 7 new ones) to evaluate Audio Large Language Models (AudioLLMs) on speech, audio scene, and voice understanding, and found that no single model performed consistently well across all tasks.
We introduce AudioBench, a universal benchmark designed to evaluate Audio Large Language Models (AudioLLMs). It encompasses 8 distinct tasks and 26 datasets, among which, 7 are newly proposed datasets. The evaluation targets three main aspects: speech understanding, audio scene understanding, and voice understanding (paralinguistic). Despite recent advancements, there lacks a comprehensive benchmark for AudioLLMs on instruction following capabilities conditioned on audio signals. AudioBench addresses this gap by setting up datasets as well as desired evaluation metrics. Besides, we also evaluated the capabilities of five popular models and found that no single model excels consistently across all tasks. We outline the research outlook for AudioLLMs and anticipate that our open-sourced evaluation toolkit, data, and leaderboard will offer a robust testbed for future model developments.