Do LLMs Need Architectural Changes for Simultaneous Speech Translation? A Prefix-to-Prefix Data Driven Approach
For researchers working on streaming speech translation, this work offers a simpler alternative to architectural modifications, though the improvement is demonstrated only on an in-house dataset.
The paper tackles simultaneous speech translation (SimulST) with decoder-only LLMs, proposing a data-driven approach (CSSEL-P2P) that uses fixed-length chunks and prefix-to-prefix targets without architectural changes, achieving a +1.54 COMETKiwi improvement over the streaming baseline at comparable latency.
Simultaneous speech translation (SimulST) requires incremental translation under strict latency constraints, yet remains challenging for decoder-only LLM systems due to limited context and cross-lingual reordering. Recent approaches often introduce architectural changes or explicit read/write policies to control output timing, which can be brittle in conversational speech where segmentation boundaries are ambiguous. We present a simple data-driven alternative: fixed-length chunks for cumulative streaming decoding with a rewind-based committed prefix, and teacher-labeled prefix-to-prefix (P2P) targets with bounded waiting for fine-tuning, yielding CSSEL-P2P, where CSSEL is our proposed chunked streaming speech encoder LLM. In our in-house conversational speech evaluation, CSSEL-P2P improves streaming quality by +1.54 COMETKiwi over the CSSEL streaming baseline at comparable latency (+0.15s Average Lagging), suggesting effective SimulST without architectural changes via P2P supervision.