Isochrony-Controlled Speech-to-Text Translation: A study on translating from Sino-Tibetan to Indo-European Languages
This work addresses the need for isochrony in speech translation applications where timing must align with audio, though it is incremental as it builds on existing sequence-to-sequence models.
The paper tackled the problem of controlling translation length to match source audio duration in speech-to-text translation, particularly for Sino-Tibetan to Indo-European languages, by improving duration alignment in a sequence-to-sequence model; it achieved 0.92 speech overlap and 8.9 BLEU on a test set, with only a 1.4 BLEU drop compared to a baseline.
End-to-end speech translation (ST), which translates source language speech directly into target language text, has garnered significant attention in recent years. Many ST applications require strict length control to ensure that the translation duration matches the length of the source audio, including both speech and pause segments. Previous methods often controlled the number of words or characters generated by the Machine Translation model to approximate the source sentence's length without considering the isochrony of pauses and speech segments, as duration can vary between languages. To address this, we present improvements to the duration alignment component of our sequence-to-sequence ST model. Our method controls translation length by predicting the duration of speech and pauses in conjunction with the translation process. This is achieved by providing timing information to the decoder, ensuring it tracks the remaining duration for speech and pauses while generating the translation. The evaluation on the Zh-En test set of CoVoST 2, demonstrates that the proposed Isochrony-Controlled ST achieves 0.92 speech overlap and 8.9 BLEU, which has only a 1.4 BLEU drop compared to the ST baseline.