ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech
This work addresses the challenge of creating realistic and stylistically diverse gestures for applications like virtual avatars or animation, though it is incremental as it builds on existing neural network frameworks for gesture generation.
The authors tackled the problem of generating gestures from speech with zero-shot style control using only a short example motion clip, even for unseen styles, and showed that their model outperforms previous state-of-the-art techniques in naturalness, appropriateness, and style portrayal in a user study.
We present ZeroEGGS, a neural network framework for speech-driven gesture generation with zero-shot style control by example. This means style can be controlled via only a short example motion clip, even for motion styles unseen during training. Our model uses a Variational framework to learn a style embedding, making it easy to modify style through latent space manipulation or blending and scaling of style embeddings. The probabilistic nature of our framework further enables the generation of a variety of outputs given the same input, addressing the stochastic nature of gesture motion. In a series of experiments, we first demonstrate the flexibility and generalizability of our model to new speakers and styles. In a user study, we then show that our model outperforms previous state-of-the-art techniques in naturalness of motion, appropriateness for speech, and style portrayal. Finally, we release a high-quality dataset of full-body gesture motion including fingers, with speech, spanning across 19 different styles.