DiffDub: Person-generic Visual Dubbing Using Inpainting Renderer with Diffusion Auto-encoder
This addresses the problem of creating realistic visual dubbing for multiple speakers and languages, though it appears incremental as it builds on a two-stage paradigm with specific improvements.
The paper tackles the challenge of generating high-quality, person-generic visual dubbing by proposing DiffDub, a diffusion-based method that uses an inpainting renderer with a mask to edit lower-face regions while preserving other parts; it outperforms existing methods with considerable margins and produces seamless, intelligible videos in person-generic and multilingual scenarios.
Generating high-quality and person-generic visual dubbing remains a challenge. Recent innovation has seen the advent of a two-stage paradigm, decoupling the rendering and lip synchronization process facilitated by intermediate representation as a conduit. Still, previous methodologies rely on rough landmarks or are confined to a single speaker, thus limiting their performance. In this paper, we propose DiffDub: Diffusion-based dubbing. We first craft the Diffusion auto-encoder by an inpainting renderer incorporating a mask to delineate editable zones and unaltered regions. This allows for seamless filling of the lower-face region while preserving the remaining parts. Throughout our experiments, we encountered several challenges. Primarily, the semantic encoder lacks robustness, constricting its ability to capture high-level features. Besides, the modeling ignored facial positioning, causing mouth or nose jitters across frames. To tackle these issues, we employ versatile strategies, including data augmentation and supplementary eye guidance. Moreover, we encapsulated a conformer-based reference encoder and motion generator fortified by a cross-attention mechanism. This enables our model to learn person-specific textures with varying references and reduces reliance on paired audio-visual data. Our rigorous experiments comprehensively highlight that our ground-breaking approach outpaces existing methods with considerable margins and delivers seamless, intelligible videos in person-generic and multilingual scenarios.