Speech-Gesture GAN: Gesture Generation for Robots and Embodied Agents
This addresses the need for nonverbal behavior in human-agent interactions, though it is incremental as it builds on existing GAN methods with a new application.
The paper tackles generating co-speech gestures for robots and embodied agents from speech text and audio, using a conditional GAN to learn relationships between gestures and speech features, with results showing efficacy in evaluations.
Embodied agents, in the form of virtual agents or social robots, are rapidly becoming more widespread. In human-human interactions, humans use nonverbal behaviours to convey their attitudes, feelings, and intentions. Therefore, this capability is also required for embodied agents in order to enhance the quality and effectiveness of their interactions with humans. In this paper, we propose a novel framework that can generate sequences of joint angles from the speech text and speech audio utterances. Based on a conditional Generative Adversarial Network (GAN), our proposed neural network model learns the relationships between the co-speech gestures and both semantic and acoustic features from the speech input. In order to train our neural network model, we employ a public dataset containing co-speech gestures with corresponding speech audio utterances, which were captured from a single male native English speaker. The results from both objective and subjective evaluations demonstrate the efficacy of our gesture-generation framework for Robots and Embodied Agents.