Hear, Invoke, and Understand: A Skill-Calling Multimodal Agent for Large Audio Language Models
This work advances audio agents by enabling them to interact with external tools and reason over observations, which is a step toward more capable AI assistants for complex acoustic problems.
The authors introduce SpeechAgent-R, an audio agent that combines intrinsic multimodal understanding with external tools for tool-interactive audio reasoning. They construct HIU-Corpus (65,492 trajectories, 507.6 hours) and HIU-Bench (1,395 samples, 56 tasks), and show that SpeechAgent-R improves over the base model by 15.40 points on in-distribution tasks and 14.23 points on out-of-distribution tasks.
Complex acoustic problems may require models to perform acoustic operations, interact with external tools and reason over the resulting textual or processed-audio observations rather than answer directly from a fixed audio input. We study such problems as tool-interactive audio reasoning and develop SpeechAgent-R, an audio agent that coordinates its intrinsic multimodal understanding with external skills and tools. To support this capability, we construct HIU-Corpus, comprising 65,492 interaction trajectories and 507.6 hours of audio across 24 tasks, 8 skills and 9 tools. SpeechAgent-R first learns structured interaction behaviors through trajectory-based supervised fine-tuning and then improves its decisions through multi-turn reinforcement learning. We further introduce HIU-Bench to jointly evaluate task performance, interaction quality and generalization to diverse task settings. It contains 1,395 samples across 56 tasks, including in-distribution (ID) and out-of-distribution (OOD) splits with substantial shifts in tool usage and workflow composition. SpeechAgent-R achieves 84.17 on ID tasks and 70.94 on OOD tasks, improving over the base model under the same agent harness by 15.40 and 14.23 points. These results demonstrate that learning skill and tool coordination improves audio agents' ability to handle diverse task settings and adaptive tool interactions.