SLIDE: Integrating Speech Language Model with LLM for Spontaneous Spoken Dialogue Generation
This work addresses the challenge of generating realistic spoken dialogue for applications like conversational AI, though it appears incremental as it combines existing SLM and LLM components.
The paper tackled the problem of generating semantically coherent spontaneous spoken dialogue by integrating a speech language model (SLM) with a large language model (LLM), resulting in a system that produces naturalistic spoken dialogue with high semantic coherence as demonstrated on the Fisher dataset.
Recently, ``textless" speech language models (SLMs) based on speech units have made huge progress in generating naturalistic speech, including non-verbal vocalizations. However, the generated speech samples often lack semantic coherence. In this paper, we propose SLM and LLM Integration for spontaneous spoken Dialogue gEneration (SLIDE). Specifically, we first utilize an LLM to generate the textual content of spoken dialogue. Next, we convert the textual dialogues into phoneme sequences and use a two-tower transformer-based duration predictor to predict the duration of each phoneme. Finally, an SLM conditioned on the spoken phoneme sequences is used to vocalize the textual dialogue. Experimental results on the Fisher dataset demonstrate that our system can generate naturalistic spoken dialogue while maintaining high semantic coherence.