Gesticulator: A framework for semantically-aware speech-driven gesture generation
This work addresses the need for more natural interactions with social agents by enabling them to generate a wider range of gestures, though it is incremental as it builds on existing gesture generation methods by combining modalities.
The paper tackles the problem of generating both beat and semantic gestures from speech by using both acoustic and semantic representations as input, resulting in a model that successfully produces realistic co-speech gestures for virtual agents and robots, as confirmed by subjective and objective evaluations.
During speech, people spontaneously gesticulate, which plays a key role in conveying information. Similarly, realistic co-speech gestures are crucial to enable natural and smooth interactions with social agents. Current end-to-end co-speech gesture generation systems use a single modality for representing speech: either audio or text. These systems are therefore confined to producing either acoustically-linked beat gestures or semantically-linked gesticulation (e.g., raising a hand when saying "high"): they cannot appropriately learn to generate both gesture types. We present a model designed to produce arbitrary beat and semantic gestures together. Our deep-learning based model takes both acoustic and semantic representations of speech as input, and generates gestures as a sequence of joint angle rotations as output. The resulting gestures can be applied to both virtual agents and humanoid robots. Subjective and objective evaluations confirm the success of our approach. The code and video are available at the project page https://svito-zar.github.io/gesticulator .