Streaming Speech-to-Text Translation with a SpeechLLM
For applications requiring real-time speech translation, this work addresses the latency bottleneck of existing SpeechLLM systems by enabling streaming output with minimal quality loss.
This work proposes an LLM-based architecture for real streaming speech-to-text translation that learns to decide when it has seen enough audio to emit output tokens. The system achieves translation quality close to the non-streaming baseline with a latency of only 1-2 seconds.
Normally, a system that translates speech into text consists of separate modules for speech recognition and text-to-text translation. Combining those tasks into a SpeechLLM promises to exploit paralinguistic information in the speech and to reduce cascaded errors. But existing SpeechLLM systems are slow since they do not work in a real streaming fashion: they wait for a complete utterance of audio before outputting a translation, or output tokens at fixed intervals, which is not suitable for real applications. This work proposes an LLM-based architecture for real streaming speech-to-text translation. The LLM learns not just to emit output tokens, but also to decide whether it has seen enough audio to do so. The system is trained using automatic alignments of the input speech and the output text. In experiments on different language pairs, the system achieves a translation quality close to the non-streaming baseline, but with a latency of only 1-2 seconds.