From Speech-to-Speech Translation to Automatic Dubbing
This work addresses the challenge of producing natural-sounding dubbed content for media localization, but it appears incremental as it builds on existing speech-to-speech translation pipelines.
The paper tackled the problem of enhancing speech-to-speech translation for automatic dubbing by integrating neural machine translation, prosodic alignment, neural text-to-speech with duration fine-tuning, and audio rendering, and reported results from a subjective evaluation on TED Talks from English to Italian.
We present enhancements to a speech-to-speech translation pipeline in order to perform automatic dubbing. Our architecture features neural machine translation generating output of preferred length, prosodic alignment of the translation with the original speech segments, neural text-to-speech with fine tuning of the duration of each utterance, and, finally, audio rendering to enriches text-to-speech output with background noise and reverberation extracted from the original audio. We report on a subjective evaluation of automatic dubbing of excerpts of TED Talks from English into Italian, which measures the perceived naturalness of automatic dubbing and the relative importance of each proposed enhancement.