VocalBench-zh: Decomposing and Benchmarking the Speech Conversational Abilities in Mandarin Context
This provides a systematic evaluation tool for developers and users in Mandarin speech interaction, though it is incremental as it adapts existing benchmarking concepts to a specific language context.
The authors tackled the lack of comprehensive speech-to-speech benchmarks in Mandarin by introducing VocalBench-zh, a dataset with over 10K instances across 10 subsets, and evaluated 14 models to reveal common challenges.
The development of multi-modal large language models (LLMs) leads to intelligent approaches capable of speech interactions. As one of the most widely spoken languages globally, Mandarin is supported by most models to enhance their applicability and reach. However, the scarcity of comprehensive speech-to-speech (S2S) benchmarks in Mandarin contexts impedes systematic evaluation for developers and hinders fair model comparison for users. In this work, we propose VocalBench-zh, an ability-level divided evaluation suite adapted to Mandarin context consisting of 10 well-crafted subsets and over 10K high-quality instances, covering 12 user-oriented characters. The evaluation experiment on 14 mainstream models reveals the common challenges for current routes, and highlights the need for new insights into next-generation speech interactive systems. The evaluation codes and datasets will be available at https://github.com/SJTU-OmniAgent/VocalBench-zh.