URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models
This addresses a gap in evaluation frameworks for spoken dialogue models, which is incremental as it builds on existing LLM advancements but introduces new speech-specific metrics.
The authors tackled the lack of comprehensive evaluation for end-to-end spoken dialogue models (SDMs) by proposing URO-Bench, a benchmark covering multilingualism, multi-round dialogues, and paralinguistics, revealing that current open-source SDMs perform well in daily QA but lag in instruction-following and paralinguistic tasks.
Recent advances in large language models (LLMs) have driven significant progress in end-to-end spoken dialogue models (SDMs). In contrast to text-based LLMs, the evaluation framework for SDMs should encompass both cognitive dimensions (e.g., logical reasoning, knowledge) and speech-related aspects (e.g., paralinguistic cues, audio quality). However, there is still a lack of comprehensive evaluations for SDMs in speech-to-speech (S2S) scenarios. To address this gap, we propose URO-Bench, an extensive benchmark for SDMs. Notably, URO-Bench is the first S2S benchmark that covers evaluations about multilingualism, multi-round dialogues, and paralinguistics. Our benchmark is divided into two difficulty levels: basic track and pro track, each comprising 20 test sets, evaluating the spoken dialogue model's abilities in Understanding, Reasoning, and Oral conversation. Evaluations on our proposed benchmark reveal that current open-source SDMs perform rather well in daily QA tasks, but lag behind their backbone LLMs in terms of instruction-following ability and also suffer from catastrophic forgetting. Their performance in advanced evaluations of paralinguistic information and audio understanding remains subpar, highlighting the need for further research in this direction. We hope that URO-Bench can facilitate the development of spoken dialogue models by providing a multifaceted evaluation of existing models and helping to track progress in this area.