FacEDiT: Unified Talking Face Editing and Generation via Facial Motion Infilling
This work addresses the lack of a unified approach for dynamic talking face synthesis, offering a method for localized edits and generation, though it is incremental in combining existing techniques like diffusion transformers and flow matching.
The authors tackled the problem of unifying talking face editing and generation by framing them as subtasks of speech-conditional facial motion infilling, resulting in FacEDiT, which produces accurate, speech-aligned edits with strong identity preservation and smooth visual continuity.
Talking face editing and face generation have often been studied as distinct problems. In this work, we propose viewing both not as separate tasks but as subtasks of a unifying formulation, speech-conditional facial motion infilling. We explore facial motion infilling as a self-supervised pretext task that also serves as a unifying formulation of dynamic talking face synthesis. To instantiate this idea, we propose FacEDiT, a speech-conditional Diffusion Transformer trained with flow matching. Inspired by masked autoencoders, FacEDiT learns to synthesize masked facial motions conditioned on surrounding motions and speech. This formulation enables both localized generation and edits, such as substitution, insertion, and deletion, while ensuring seamless transitions with unedited regions. In addition, biased attention and temporal smoothness constraints enhance boundary continuity and lip synchronization. To address the lack of a standard editing benchmark, we introduce FacEDiTBench, the first dataset for talking face editing, featuring diverse edit types and lengths, along with new evaluation metrics. Extensive experiments validate that talking face editing and generation emerge as subtasks of speech-conditional motion infilling; FacEDiT produces accurate, speech-aligned facial edits with strong identity preservation and smooth visual continuity while generalizing effectively to talking face generation.