Towards human-like spoken dialogue generation between AI agents from written dialogue
This work addresses the problem of creating realistic spoken interactions for AI agents, which is incremental as it builds on existing text-to-speech methods by incorporating features like backchannels and turn-taking.
The study tackled the challenge of generating human-like spoken dialogues from written dialogues by proposing CHATS, a discrete token-based system that simultaneously generates speech for both speaker and listener sides using only speaker transcriptions, and it outperformed the text-to-speech baseline in producing more interactive and fluid dialogues while maintaining clarity and intelligibility.
The advent of large language models (LLMs) has made it possible to generate natural written dialogues between two agents. However, generating human-like spoken dialogues from these written dialogues remains challenging. Spoken dialogues have several unique characteristics: they frequently include backchannels and laughter, and the smoothness of turn-taking significantly influences the fluidity of conversation. This study proposes CHATS - CHatty Agents Text-to-Speech - a discrete token-based system designed to generate spoken dialogues based on written dialogues. Our system can generate speech for both the speaker side and the listener side simultaneously, using only the transcription from the speaker side, which eliminates the need for transcriptions of backchannels or laughter. Moreover, CHATS facilitates natural turn-taking; it determines the appropriate duration of silence after each utterance in the absence of overlap, and it initiates the generation of overlapping speech based on the phoneme sequence of the next utterance in case of overlap. Experimental evaluations indicate that CHATS outperforms the text-to-speech baseline, producing spoken dialogues that are more interactive and fluid while retaining clarity and intelligibility.