C2G2: Controllable Co-speech Gesture Generation with Latent Diffusion Model
This work improves digital avatar animation by enabling more stable and controllable gesture generation, though it appears incremental as it builds on existing latent diffusion models.
The paper tackles co-speech gesture generation for digital avatars by addressing issues like unstable training and lack of control, proposing C2G2 with a two-stage temporal dependency enhancement and features for speaker identity and editing, achieving state-of-the-art results on benchmark datasets.
Co-speech gesture generation is crucial for automatic digital avatar animation. However, existing methods suffer from issues such as unstable training and temporal inconsistency, particularly in generating high-fidelity and comprehensive gestures. Additionally, these methods lack effective control over speaker identity and temporal editing of the generated gestures. Focusing on capturing temporal latent information and applying practical controlling, we propose a Controllable Co-speech Gesture Generation framework, named C2G2. Specifically, we propose a two-stage temporal dependency enhancement strategy motivated by latent diffusion models. We further introduce two key features to C2G2, namely a speaker-specific decoder to generate speaker-related real-length skeletons and a repainting strategy for flexible gesture generation/editing. Extensive experiments on benchmark gesture datasets verify the effectiveness of our proposed C2G2 compared with several state-of-the-art baselines. The link of the project demo page can be found at https://c2g2-gesture.github.io/c2_gesture