Co-Speech Gesture Synthesis using Discrete Gesture Token Learning
This work solves the problem of creating believable robot motions for improved human interaction in applications like education and healthcare, but it is incremental as it builds on existing gesture synthesis methods.
The paper tackled the problem of synthesizing realistic co-speech gestures for humanoid robots by addressing the uncertainty of multiple viable motions for the same speech, resulting in a method that outperforms previous approaches in generating realistic and diverse gestures as shown in quantitative results and user studies.
Synthesizing realistic co-speech gestures is an important and yet unsolved problem for creating believable motions that can drive a humanoid robot to interact and communicate with human users. Such capability will improve the impressions of the robots by human users and will find applications in education, training, and medical services. One challenge in learning the co-speech gesture model is that there may be multiple viable gesture motions for the same speech utterance. The deterministic regression methods can not resolve the conflicting samples and may produce over-smoothed or damped motions. We proposed a two-stage model to address this uncertainty issue in gesture synthesis by modeling the gesture segments as discrete latent codes. Our method utilizes RQ-VAE in the first stage to learn a discrete codebook consisting of gesture tokens from training data. In the second stage, a two-level autoregressive transformer model is used to learn the prior distribution of residual codes conditioned on input speech context. Since the inference is formulated as token sampling, multiple gesture sequences could be generated given the same speech input using top-k sampling. The quantitative results and the user study showed the proposed method outperforms the previous methods and is able to generate realistic and diverse gesture motions.