UniSync: Towards Generalizable and High-Fidelity Lip Synchronization for Challenging Scenarios
This work addresses the challenge of high-fidelity lip synchronization for video dubbing in diverse real-world scenarios, representing a strong specific gain rather than a foundational advancement.
The paper tackles the problem of generating realistic talking videos that match given audio by addressing limitations of current methods in handling diverse real-world scenarios like stylized avatars and extreme lighting. The result is UniSync, a framework that significantly outperforms state-of-the-art methods, as demonstrated through extensive experiments on a new benchmark.
Lip synchronization aims to generate realistic talking videos that match given audio, which is essential for high-quality video dubbing. However, current methods have fundamental drawbacks: mask-based approaches suffer from local color discrepancies, while mask-free methods struggle with global background texture misalignment. Furthermore, most methods struggle with diverse real-world scenarios such as stylized avatars, face occlusion, and extreme lighting conditions. In this paper, we propose UniSync, a unified framework designed for achieving high-fidelity lip synchronization in diverse scenarios. Specifically, UniSync uses a mask-free pose-anchored training strategy to keep head motion and eliminate synthesis color artifacts, while employing mask-based blending consistent inference to ensure structural precision and smooth blending. Notably, fine-tuning on compact but diverse videos empowers our model with exceptional domain adaptability, handling complex corner cases effectively. We also introduce the RealWorld-LipSync benchmark to evaluate models under real-world demands, which covers diverse application scenarios including both human faces and stylized avatars. Extensive experiments demonstrate that UniSync significantly outperforms state-of-the-art methods, advancing the field towards truly generalizable and production-ready lip synchronization.