Decision Attentive Regularization to Improve Simultaneous Speech Translation Systems
This work addresses the challenge of real-time speech translation for applications requiring low-latency output, though it is incremental by extending existing techniques from offline domains.
The paper tackled the problem of improving simultaneous speech-to-text translation (SimulST) by leveraging text transcripts to enhance decision policies, achieving a 34.66% or 4.5 BLEU improvement over the baseline across different latency regimes for the English-German task.
Simultaneous translation systems start producing the output while processing the partial source sentence in the incoming input stream. These systems need to decide when to read more input and when to write the output. These decisions depend on the structure of source/target language and the information contained in the partial input sequence. Hence, read/write decision policy remains the same across different input modalities, i.e., speech and text. This motivates us to leverage the text transcripts corresponding to the speech input for improving simultaneous speech-to-text translation (SimulST). We propose Decision Attentive Regularization (DAR) to improve the decision policy of SimulST systems by using the simultaneous text-to-text translation (SimulMT) task. We also extend several techniques from the offline speech translation domain to explore the role of SimulMT task in improving SimulST performance. Overall, we achieve 34.66% / 4.5 BLEU improvement over the baseline model across different latency regimes for the MuST-C English-German (EnDe) SimulST task.