Chain-of-Thought Training for Open E2E Spoken Dialogue Systems
This addresses the challenge of building efficient and coherent spoken dialogue systems for conversational AI applications, though it appears incremental.
The paper tackles the problem of end-to-end spoken dialogue systems requiring large training data and generating incoherent responses by proposing a chain-of-thought training strategy, achieving over 1.5 ROUGE-1 improvement and enabling training on just 300 hours of data.
Unlike traditional cascaded pipelines, end-to-end (E2E) spoken dialogue systems preserve full differentiability and capture non-phonemic information, making them well-suited for modeling spoken interactions. However, existing E2E approaches often require large-scale training data and generates responses lacking semantic coherence. We propose a simple yet effective strategy leveraging a chain-of-thought (CoT) formulation, ensuring that training on conversational data remains closely aligned with the multimodal language model (LM)'s pre-training on speech recognition~(ASR), text-to-speech synthesis (TTS), and text LM tasks. Our method achieves over 1.5 ROUGE-1 improvement over the baseline, successfully training spoken dialogue systems on publicly available human-human conversation datasets, while being compute-efficient enough to train on just 300 hours of public human-human conversation data, such as the Switchboard. We will publicly release our models and training code.