NaturalFlow: Reducing Disruptive Pauses for Natural Speech Flow in Simultaneous Speech-to-Speech Translation
For users of real-time speech translation, this work addresses the problem of unnatural pauses that increase cognitive load, offering a more natural listening experience without sacrificing latency.
Simultaneous speech-to-speech translation often produces fragmented speech with unnatural pauses. The authors propose a fluency-aware optimization framework that reduces inter-chunk silences by leveraging model-internal signals, achieving natural speech flow while maintaining competitive latency and translation quality.
Simultaneous speech-to-speech translation aims to enable near-real-time communication by minimizing latency, offering a compelling, real-time alternative to the high latency of consecutive translation. However, the excessive pursuit of low latency often results in fragmented chunk-wise speech. Consequently, listeners are subjected to an unnatural acoustic flow punctuated by frequent pauses, which could increase their cognitive load. To bridge this gap, we introduce a fluency-aware optimization framework designed to discover the sweet spot between the low-latency benefits of simultaneous translation and the natural flow of consecutive translation. Our framework minimizes inter-chunk silences by leveraging model-internal signals, including linguistic diversity and induced temporal variability in speech durations. Experiments on short- and long-form benchmarks show that our framework produces natural speech flow while maintaining competitive latency and translation quality.