Self-Supervised Learning of Deviation in Latent Representation for Co-speech Gesture Video Generation
This addresses the challenge of enhancing co-speech communication through realistic gesture generation, representing an incremental advance over existing methods.
The paper tackles the problem of generating realistic co-speech gesture videos by exploring self-supervised representation and pixel-level motion deviation using a diffusion model with latent motion features, resulting in improvements of 2.7-4.5% for FGD, DIV, and FVD, and 8.1% for PSNR and 2.5% for SSIM over state-of-the-art methods.
Gestures are pivotal in enhancing co-speech communication. While recent works have mostly focused on point-level motion transformation or fully supervised motion representations through data-driven approaches, we explore the representation of gestures in co-speech, with a focus on self-supervised representation and pixel-level motion deviation, utilizing a diffusion model which incorporates latent motion features. Our approach leverages self-supervised deviation in latent representation to facilitate hand gestures generation, which are crucial for generating realistic gesture videos. Results of our first experiment demonstrate that our method enhances the quality of generated videos, with an improvement from 2.7 to 4.5% for FGD, DIV, and FVD, and 8.1% for PSNR, 2.5% for SSIM over the current state-of-the-art methods.