SpokenUS: A Spoken User Simulator for Task-Oriented Dialogue
This addresses the problem of training robust spoken dialogue agents for task-oriented applications by providing a scalable dataset and simulator, though it is incremental in improving existing simulation methods.
The authors tackled the lack of large-scale spoken task-oriented dialogue data by introducing SpokenTOD, a dataset with 52,390 dialogues and 1,034 hours of speech, and SpokenUS, a spoken user simulator that achieves comparable goal coverage to larger models and outperforms baselines in Human MOS.
Robust task-oriented spoken dialogue agents require exposure to the full diversity of how people interact through speech. Building spoken user simulators that address this requires large-scale spoken task-oriented dialogue (TOD) data encompassing spoken user behaviors, yet existing datasets are limited in scale and domain coverage, with no systematic pipeline for augmenting them. To address this, we introduce \textbf{SpokenTOD}, a spoken TOD dataset of 52,390 dialogues and 1,034 hours of speech augmented with four spoken user behaviors -- cross-turn slots, barge-in, disfluency, and emotional prosody -- across diverse speakers and domains. Building on SpokenTOD, we present \textbf{SpokenUS}, a spoken user simulator grounded in TOD with a dedicated architecture for barge-in. SpokenUS achieves comparable goal coverage to significantly larger models while substantially outperforming all baselines in Human MOS, disclosing slot values gradually across the dialogue as humans do rather than front-loading them. Further analysis confirms that SpokenUS's spoken behaviors pose meaningful challenges to downstream agents, making it a practical tool for training and evaluating more robust spoken dialogue systems.