Duplex Diffusion Models Improve Speech-to-Speech Translation
This work addresses the challenge of efficient and high-performance bidirectional speech translation for applications like real-time communication, offering a novel solution that improves over existing separate or multitask models.
The paper tackles the problem of bidirectional speech-to-speech translation by proposing a duplex diffusion model that integrates diffusion probabilistic models with a reversible Conformer, enabling reversible translation by flipping input and output ends. It achieves significant improvements in ASR-BLEU scores compared to state-of-the-art baselines, marking the first success in reversible speech translation.
Speech-to-speech translation is a typical sequence-to-sequence learning task that naturally has two directions. How to effectively leverage bidirectional supervision signals to produce high-fidelity audio for both directions? Existing approaches either train two separate models or a multitask-learned model with low efficiency and inferior performance. In this paper, we propose a duplex diffusion model that applies diffusion probabilistic models to both sides of a reversible duplex Conformer, so that either end can simultaneously input and output a distinct language's speech. Our model enables reversible speech translation by simply flipping the input and output ends. Experiments show that our model achieves the first success of reversible speech translation with significant improvements of ASR-BLEU scores compared with a list of state-of-the-art baselines.