StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generator
This addresses the challenge of producing realistic and personalized lip movements for applications like video generation, though it appears incremental by building on style-based generators.
The paper tackles the problem of balancing generation quality and generalization ability in lip synchronization with audio, proposing StyleSync to achieve high-fidelity results in one-shot and few-shot scenarios, with experiments demonstrating effectiveness across various scenes.
Despite recent advances in syncing lip movements with any audio waves, current methods still struggle to balance generation quality and the model's generalization ability. Previous studies either require long-term data for training or produce a similar movement pattern on all subjects with low quality. In this paper, we propose StyleSync, an effective framework that enables high-fidelity lip synchronization. We identify that a style-based generator would sufficiently enable such a charming property on both one-shot and few-shot scenarios. Specifically, we design a mask-guided spatial information encoding module that preserves the details of the given face. The mouth shapes are accurately modified by audio through modulated convolutions. Moreover, our design also enables personalized lip-sync by introducing style space and generator refinement on only limited frames. Thus the identity and talking style of a target person could be accurately preserved. Extensive experiments demonstrate the effectiveness of our method in producing high-fidelity results on a variety of scenes. Resources can be found at https://hangz-nju-cuhk.github.io/projects/StyleSync.