VisualTTS: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Over
This addresses the need for automated voice-over generation in video production, offering a domain-specific solution for media and entertainment applications.
The paper tackles the problem of synthesizing speech synchronized with lip movements in silent videos, known as automatic voice over (AVO), and proposes VisualTTS, a model that achieves accurate lip-speech synchronization and outperforms baseline systems.
In this paper, we formulate a novel task to synthesize speech in sync with a silent pre-recorded video, denoted as automatic voice over (AVO). Unlike traditional speech synthesis, AVO seeks to generate not only human-sounding speech, but also perfect lip-speech synchronization. A natural solution to AVO is to condition the speech rendering on the temporal progression of lip sequence in the video. We propose a novel text-to-speech model that is conditioned on visual input, named VisualTTS, for accurate lip-speech synchronization. The proposed VisualTTS adopts two novel mechanisms that are 1) textual-visual attention, and 2) visual fusion strategy during acoustic decoding, which both contribute to forming accurate alignment between the input text content and lip motion in input lip sequence. Experimental results show that VisualTTS achieves accurate lip-speech synchronization and outperforms all baseline systems.