Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agents
This addresses the need for interactive, emotionally expressive virtual agents in applications like narration or conversation, though it is incremental as it builds on existing transformer-based methods.
The authors tackled the problem of generating emotive full-body gestures for virtual agents from natural language text, achieving state-of-the-art performance with 91% of participants rating gestures as plausible and strong emotional correlation (minimum Pearson coefficient 0.77).
We present Text2Gestures, a transformer-based learning method to interactively generate emotive full-body gestures for virtual agents aligned with natural language text inputs. Our method generates emotionally expressive gestures by utilizing the relevant biomechanical features for body expressions, also known as affective features. We also consider the intended task corresponding to the text and the target virtual agents' intended gender and handedness in our generation pipeline. We train and evaluate our network on the MPI Emotional Body Expressions Database and observe that our network produces state-of-the-art performance in generating gestures for virtual agents aligned with the text for narration or conversation. Our network can generate these gestures at interactive rates on a commodity GPU. We conduct a web-based user study and observe that around 91% of participants indicated our generated gestures to be at least plausible on a five-point Likert Scale. The emotions perceived by the participants from the gestures are also strongly positively correlated with the corresponding intended emotions, with a minimum Pearson coefficient of 0.77 in the valence dimension.