Generative Spoken Dialogue Language Modeling
This addresses the problem of generating realistic spoken dialogues for applications like virtual assistants or conversational AI, though it builds on existing methods for unit discovery and transformer architectures.
The researchers tackled the problem of generating naturalistic spoken dialogues without text by introducing dGSLM, a textless model that uses unsupervised spoken unit discovery and a dual-tower transformer trained on 2000 hours of raw conversational audio. The result showed it generates speech, laughter, and paralinguistic signals in two channels simultaneously and produces more naturalistic turn-taking compared to text-based models.
We introduce dGSLM, the first "textless" model able to generate audio samples of naturalistic spoken dialogues. It uses recent work on unsupervised spoken unit discovery coupled with a dual-tower transformer architecture with cross-attention trained on 2000 hours of two-channel raw conversational audio (Fisher dataset) without any text or labels. We show that our model is able to generate speech, laughter and other paralinguistic signals in the two channels simultaneously and reproduces more naturalistic and fluid turn-taking compared to a text-based cascaded model.