KSDiff: Keyframe-Augmented Speech-Aware Dual-Path Diffusion for Facial Animation
This work addresses the challenge of generating realistic talking-head animations for multimedia applications, representing an incremental improvement over existing methods.
The paper tackled the problem of audio-driven facial animation by proposing KSDiff, a framework that disentangles speech features and models keyframes to improve lip synchronization and head-pose naturalness, achieving state-of-the-art performance on HDTF and VoxCeleb datasets.
Audio-driven facial animation has made significant progress in multimedia applications, with diffusion models showing strong potential for talking-face synthesis. However, most existing works treat speech features as a monolithic representation and fail to capture their fine-grained roles in driving different facial motions, while also overlooking the importance of modeling keyframes with intense dynamics. To address these limitations, we propose KSDiff, a Keyframe-Augmented Speech-Aware Dual-Path Diffusion framework. Specifically, the raw audio and transcript are processed by a Dual-Path Speech Encoder (DPSE) to disentangle expression-related and head-pose-related features, while an autoregressive Keyframe Establishment Learning (KEL) module predicts the most salient motion frames. These components are integrated into a Dual-path Motion generator to synthesize coherent and realistic facial motions. Extensive experiments on HDTF and VoxCeleb demonstrate that KSDiff achieves state-of-the-art performance, with improvements in both lip synchronization accuracy and head-pose naturalness. Our results highlight the effectiveness of combining speech disentanglement with keyframe-aware diffusion for talking-head generation.