Fast When, Careful Who: Dual-Process Multiparty Turn-Taking with Diffusion Augmentation
For developers of multiparty spoken dialogue systems, this work addresses the challenge of handling overlapping speech and rapid speaker changes, though improvements are incremental.
The paper tackles multiparty turn-taking in spoken dialogue systems, proposing a two-stage pipeline that separates turn-boundary detection from floor-transfer verification. On the VoxConverse dataset, the method improves shift detection over a baseline, with additional gains from diffusion-based data augmentation.
Reliable turn-taking is essential for spoken dialogue systems. However, most existing methods are designed for two-speaker interaction and struggle with realistic multiparty audio containing overlap and rapid speaker changes. We study multiparty turn-taking on the VoxConverse dataset and propose an audio-only two-stage pipeline that separates when to trigger a turn boundary from whether the floor is actually transferring. A fast trigger scans the audio and proposes candidate end-of-turn times, while a lightweight verifier runs only at those times to decide \textsc{Hold} or \textsc{Shift} and support next-speaker prediction. We report results in the full multiparty setting and a controlled dyadic top-2 projection for comparability. We also investigate diffusion-based, label-preserving background-audio mixing as a data augmentation strategy. Results show improved shift detection over a baseline, with further improvements from diffusion augmentation.