Video Editing for Audio-Visual Dubbing
This work addresses visual dubbing for making content accessible across languages, offering a novel approach that outperforms existing methods, though it is incremental in improving specific bottlenecks.
The paper tackles the problem of visual dubbing by introducing EdiDub, a framework that reformulates it as content-aware editing to preserve original video context, resulting in significant improvements in identity preservation and synchronization on benchmarks, with human evaluations confirming higher scores for synchronization and visual naturalness.
Visual dubbing, the synchronization of facial movements with new speech, is crucial for making content accessible across different languages, enabling broader global reach. However, current methods face significant limitations. Existing approaches often generate talking faces, hindering seamless integration into original scenes, or employ inpainting techniques that discard vital visual information like partial occlusions and lighting variations. This work introduces EdiDub, a novel framework that reformulates visual dubbing as a content-aware editing task. EdiDub preserves the original video context by utilizing a specialized conditioning scheme to ensure faithful and accurate modifications rather than mere copying. On multiple benchmarks, including a challenging occluded-lip dataset, EdiDub significantly improves identity preservation and synchronization. Human evaluations further confirm its superiority, achieving higher synchronization and visual naturalness scores compared to the leading methods. These results demonstrate that our content-aware editing approach outperforms traditional generation or inpainting, particularly in maintaining complex visual elements while ensuring accurate lip synchronization.