ExpGest: Expressive Speaker Generation Using Diffusion Model and Hybrid Audio-Text Guidance
This addresses the need for more realistic and emotionally expressive gestures in virtual speakers or avatars, representing a novel approach rather than an incremental improvement.
The paper tackles the problem of generating stiff and mechanical gestures in speaker animation by introducing ExpGest, a framework that uses synchronized text and audio information to produce expressive full-body gestures, achieving more expressive, natural, and controllable motion compared to state-of-the-art models.
Existing gesture generation methods primarily focus on upper body gestures based on audio features, neglecting speech content, emotion, and locomotion. These limitations result in stiff, mechanical gestures that fail to convey the true meaning of audio content. We introduce ExpGest, a novel framework leveraging synchronized text and audio information to generate expressive full-body gestures. Unlike AdaIN or one-hot encoding methods, we design a noise emotion classifier for optimizing adversarial direction noise, avoiding melody distortion and guiding results towards specified emotions. Moreover, aligning semantic and gestures in the latent space provides better generalization capabilities. ExpGest, a diffusion model-based gesture generation framework, is the first attempt to offer mixed generation modes, including audio-driven gestures and text-shaped motion. Experiments show that our framework effectively learns from combined text-driven motion and audio-induced gesture datasets, and preliminary results demonstrate that ExpGest achieves more expressive, natural, and controllable global motion in speakers compared to state-of-the-art models.