Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding
This work addresses the problem of efficient and accurate speech-to-text translation for applications requiring low latency and reduced error propagation, representing an incremental advance over existing multi-task or two-stage methods.
The paper tackles the challenge of implementing end-to-end speech-to-text translation without intermediate transcriptions by proposing a novel interactive attention mechanism that enables synchronous and interactive processing of automatic speech recognition and translation in a single model. Experiments on TED corpora show it outperforms strong baselines in translation quality and improves speech recognition performance.
Speech-to-text translation (ST), which translates source language speech into target language text, has attracted intensive attention in recent years. Compared to the traditional pipeline system, the end-to-end ST model has potential benefits of lower latency, smaller model size, and less error propagation. However, it is notoriously difficult to implement such a model without transcriptions as intermediate. Existing works generally apply multi-task learning to improve translation quality by jointly training end-to-end ST along with automatic speech recognition (ASR). However, different tasks in this method cannot utilize information from each other, which limits the improvement. Other works propose a two-stage model where the second model can use the hidden state from the first one, but its cascade manner greatly affects the efficiency of training and inference process. In this paper, we propose a novel interactive attention mechanism which enables ASR and ST to perform synchronously and interactively in a single model. Specifically, the generation of transcriptions and translations not only relies on its previous outputs but also the outputs predicted in the other task. Experiments on TED speech translation corpora have shown that our proposed model can outperform strong baselines on the quality of speech translation and achieve better speech recognition performances as well.