Spoken Function Calling: A New Perspective on Spoken Language Understanding for Large Audio Language Models
This work addresses the challenge of open-domain semantic understanding for large audio language models, which is crucial for improving human-agent interaction in task-oriented dialogue systems.
This paper introduces Spoken Function Calling (SFC), a new approach to spoken language understanding that uses structured rule definitions to improve semantic extraction. The authors created a new dataset, SFC-Bench, and showed that SFC significantly enhances semantic extraction accuracy for both Large Language Models (LLMs) and Large Audio Language Models (LALMs) compared to traditional SLU.
Spoken Language Understanding (SLU) is the core component of task-oriented dialogue systems and a pivotal link in achieving seamless human-agent interaction. While traditional SLU can effectively extract user semantics for closed-set tasks after in-domain supervised fine-tuning, it faces significant challenges in leveraging in-context learning for open-domain tasks due to its ambiguous rule definitions. This work proposes Spoken Function Calling (SFC), a novel semantic understanding perspective that optimizes semantic understanding with structured rule definitions, to evolve beyond traditional closed-set SLU. Specifically, we curate and extend a suite of spoken functions based on traditional SLU datasets, construct a multi-agent system to synthesize the SFC-Bench dataset, evaluate the performance of Large Language Models (LLMs) and Large Audio Language Models (LALMs), and enhance the SFC capabilities of LALMs through post-training. Experiments demonstrate that SFC outperforms traditional SLU, substantially enhancing the semantic extraction accuracy for LLMs and LALMs.