VoiceAgentBench: Are Voice Assistants ready for agentic tasks?
This addresses the problem of evaluating realistic agentic capabilities in voice assistants for researchers and developers, though it is incremental as it builds on existing benchmarking efforts.
The authors tackled the lack of systematic evaluation for voice assistants in agentic tasks by introducing VoiceAgentBench, a comprehensive benchmark with over 5,500 synthetic spoken queries across multiple languages, revealing significant gaps in contextual tool orchestration, Indic generalization, and adversarial robustness in current SpeechLMs.
Large-scale Speech Language Models (SpeechLMs) have enabled voice assistants capable of understanding natural spoken queries and performing complex tasks. However, existing speech benchmarks primarily focus on isolated capabilities such as transcription, or question-answering, and do not systematically evaluate agentic scenarios encompassing multilingual and cultural understanding, as well as adversarial robustness. To address this, we introduce VoiceAgentBench, a comprehensive benchmark designed to evaluate SpeechLMs in realistic spoken agentic settings. It comprises over 5,500 synthetic spoken queries, including dialogues grounded in Indian context, covering single-tool invocations, multi-tool workflows, multi-turn interactions, and safety evaluations. The benchmark supports English, Hindi, and 5 other Indian languages, reflecting real-world linguistic and cultural diversity. We simulate speaker variability using a novel sampling algorithm that selects audios for TTS voice conversion based on its speaker embeddings, maximizing acoustic and speaker diversity. Our evaluation measures tool selection accuracy, structural consistency, and the correctness of tool invocations, including adversarial robustness. Our experiments reveal significant gaps in contextual tool orchestration tasks, Indic generalization, and adversarial robustness, exposing critical limitations of current SpeechLMs.