MOSS-Speech: Towards True Speech-to-Speech Models Without Text Guidance
This work addresses the bottleneck in end-to-end speech interaction for applications like dialogue systems by establishing a new paradigm for expressive and efficient speech-to-speech models.
The paper tackled the problem of spoken dialogue systems relying on text intermediates, which discard paralinguistic cues and limit expressivity, by introducing MOSS-Speech, a true speech-to-speech large language model that directly understands and generates speech without text guidance, achieving state-of-the-art results in spoken question answering and comparable performance to text-guided systems.
Spoken dialogue systems often rely on cascaded pipelines that transcribe, process, and resynthesize speech. While effective, this design discards paralinguistic cues and limits expressivity. Recent end-to-end methods reduce latency and better preserve these cues, yet still rely on text intermediates, creating a fundamental bottleneck. We present MOSS-Speech, a true speech-to-speech large language model that directly understands and generates speech without relying on text guidance. Our approach combines a modality-based layer-splitting architecture with a frozen pre-training strategy, preserving the reasoning and knowledge of pretrained text LLMs while adding native speech capabilities. Experiments show that our model achieves state-of-the-art results in spoken question answering and delivers comparable speech-to-speech performance relative to existing text-guided systems, while still maintaining competitive text performance. By narrowing the gap between text-guided and direct speech generation, our work establishes a new paradigm for expressive and efficient end-to-end speech interaction.