Audio2Tool: Bridging Spoken Language Understanding and Function Calling
Provides a much-needed benchmark for evaluating tool-calling capabilities of SpeechLMs in realistic, multi-domain settings, highlighting critical failure modes for developers.
Audio2Tool introduces a large-scale benchmark of ~30,000 queries to evaluate tool-calling in SpeechLMs across Smart Car, Smart Home, and Wearables domains, revealing that current models perform well on simple commands but degrade significantly under compositional and acoustic challenges.
Voice assistants increasingly rely on Speech Language Models (SpeechLMs) to interpret spoken queries and execute complex tasks, yet existing benchmarks lack domain breadth, acoustic diversity, and compositional reasoning complexity to evaluate tool-calling performance. We introduce Audio2Tool, a large-scale dataset comprising approximately 30,000 queries designed to assess tool-calling capabilities of SpeechLMs across three primary domains: Smart Car, Smart Home, and Wearables. Our benchmark features a multi-tier complexity hierarchy, ranging from simple direct commands to complex multi-intent and needle-in-a-haystack extraction to isolate distinct failure modes. To ensure realism, we employ zero-shot voice cloning text-to-speech synthesis and diverse noise profiles to simulate in-the-wild conditions. Evaluations of state-of-the-art SpeechLMs and ASR-LLM pipelines show strong performance on simple commands but significant degradation under compositional and acoustic challenges. We will release the dataset and benchmark upon acceptance.