DeepGesture: A conversational gesture synthesis system based on emotions and semantics
This work addresses the problem of generating expressive co-speech gestures for digital humans, representing an incremental advancement in multimodal synthesis.
The paper tackles the bottleneck in creating digital humans by generating natural character movements from multimodal inputs, presenting DeepGesture, a diffusion-based gesture synthesis framework that improves semantic alignment and emotional expressiveness, achieving enhanced human-likeness and contextual appropriateness on the ZeroEGGS dataset.
Along with the explosion of large language models, improvements in speech synthesis, advancements in hardware, and the evolution of computer graphics, the current bottleneck in creating digital humans lies in generating character movements that correspond naturally to text or speech inputs. In this work, we present DeepGesture, a diffusion-based gesture synthesis framework for generating expressive co-speech gestures conditioned on multimodal signals - text, speech, emotion, and seed motion. Built upon the DiffuseStyleGesture model, DeepGesture introduces novel architectural enhancements that improve semantic alignment and emotional expressiveness in generated gestures. Specifically, we integrate fast text transcriptions as semantic conditioning and implement emotion-guided classifier-free diffusion to support controllable gesture generation across affective states. To visualize results, we implement a full rendering pipeline in Unity based on BVH output from the model. Evaluation on the ZeroEGGS dataset shows that DeepGesture produces gestures with improved human-likeness and contextual appropriateness. Our system supports interpolation between emotional states and demonstrates generalization to out-of-distribution speech, including synthetic voices - marking a step forward toward fully multimodal, emotionally aware digital humans. Project page: https://deepgesture.github.io