Isochrony-Aware Neural Machine Translation for Automatic Dubbing
This addresses the challenge of automatic dubbing for media localization, though it is incremental as it builds on existing neural machine translation methods.
The paper tackled the problem of generating translations suitable for dubbing by ensuring audiovisual coherence through isochrony (matching speech-pause structure and duration), and found that the simplest modeling approach worked best in experiments on English-German/French language pairs, with results confirmed by human evaluations.
We introduce the task of isochrony-aware machine translation which aims at generating translations suitable for dubbing. Dubbing of a spoken sentence requires transferring the content as well as the speech-pause structure of the source into the target language to achieve audiovisual coherence. Practically, this implies correctly projecting pauses from the source to the target and ensuring that target speech segments have roughly the same duration of the corresponding source speech segments. In this work, we propose implicit and explicit modeling approaches to integrate isochrony information into neural machine translation. Experiments on English-German/French language pairs with automatic metrics show that the simplest of the considered approaches works best. Results are confirmed by human evaluations of translations and dubbed videos.