Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs
This work addresses the effectiveness of speech modality integration in LLMs for speech translation, providing a comprehensive benchmark for researchers and practitioners, but it is incremental as it compares existing methods without introducing new techniques.
The study evaluated whether integrating speech directly into Large Language Models (SpeechLLMs) improves speech-to-text translation quality compared to traditional cascaded systems, finding that cascaded systems remain the most reliable overall, with SpeechLLMs only matching them in selected settings.
As Large Language Models (LLMs) expand beyond text, integrating speech as a native modality has given rise to SpeechLLMs, which aim to translate spoken language directly, thereby bypassing traditional transcription-based pipelines. Whether this integration improves speech-to-text translation quality over established cascaded architectures, however, remains an open question. We present Hearing to Translate, the first comprehensive test suite rigorously benchmarking 5 state-of-the-art SpeechLLMs against 16 strong direct and cascade systems that couple leading speech foundation models (SFM), with multilingual LLMs. Our analysis spans 16 benchmarks, 13 language pairs, and 9 challenging conditions, including disfluent, noisy, and long-form speech. Across this extensive evaluation, we find that cascaded systems remain the most reliable overall, while current SpeechLLMs only match cascades in selected settings and SFMs lag behind both, highlighting that integrating an LLM, either within the model or in a pipeline, is essential for high-quality speech translation.