Streaming T5-based Text-to-Speech Synthesis with Limited Lookahead
Reduces latency in cascaded LLM-TTS systems for real-time conversational AI applications.
S5-TTS enables streaming text-to-speech by generating speech word-by-word with limited lookahead, achieving comparable quality to full-context T5-TTS while significantly reducing end-to-end latency for conversational AI.
Streaming text-to-speech synthesis in cascaded LLM-TTS systems still faces latency challenges as most TTS models require full context before initiating generation. We present S5-TTS, a streaming variant of T5-TTS that enables low-latency, word-by-word incremental speech synthesis through encoder-decoder language modeling and monotonic alignment learning. S5-TTS begins generating speech immediately after receiving the first few words, substantially reducing end-to-end response latency. To maintain quality under limited lookahead, we introduce a lookahead-causal masking mechanism with Conv-based auxiliary attention that preserves intelligibility and speaker similarity, and employ interleaved multi-source distillation to further restore naturalness. Experiments show that S5-TTS achieves comparable quality to full-context T5-TTS, supports zero-shot synthesis with high speaker similarity, and significantly reduces end-to-end latency for practical conversational AI systems.