Spoken Conversational Agents with Large Language Models
This is an incremental tutorial for researchers and practitioners in conversational AI, focusing on practical recipes and a roadmap for developing more advanced spoken agents.
The tutorial addresses the transition of spoken conversational agents from cascaded ASR/NLU systems to voice-native large language models, covering adaptation techniques, cross-modal alignment, and joint training, while reviewing datasets, metrics, and robustness across accents.
Spoken conversational agents are converging toward voice-native LLMs. This tutorial distills the path from cascaded ASR/NLU to end-to-end, retrieval-and vision-grounded systems. We frame adaptation of text LLMs to audio, cross-modal alignment, and joint speech-text training; review datasets, metrics, and robustness across accents and compare design choices (cascaded vs. E2E, post-ASR correction, streaming). We link industrial assistants to current open-domain and task-oriented agents, highlight reproducible baselines, and outline open problems in privacy, safety, and evaluation. Attendees leave with practical recipes and a clear systems-level roadmap.