MPE4G: Multimodal Pretrained Encoder for Co-Speech Gesture Generation
This work addresses the need for more reliable and generalized gesture generation in human-agent interactions, representing an incremental improvement over previous models.
The paper tackles the problem of generating realistic co-speech gestures for virtual agents, especially when input modalities are missing or noisy, by proposing a multimodal pre-trained encoder framework, which achieves robust gesture generation validated through experiments and human evaluation.
When virtual agents interact with humans, gestures are crucial to delivering their intentions with speech. Previous multimodal co-speech gesture generation models required encoded features of all modalities to generate gestures. If some input modalities are removed or contain noise, the model may not generate the gestures properly. To acquire robust and generalized encodings, we propose a novel framework with a multimodal pre-trained encoder for co-speech gesture generation. In the proposed method, the multi-head-attention-based encoder is trained with self-supervised learning to contain the information on each modality. Moreover, we collect full-body gestures that consist of 3D joint rotations to improve visualization and apply gestures to the extensible body model. Through the series of experiments and human evaluation, the proposed method renders realistic co-speech gestures not only when all input modalities are given but also when the input modalities are missing or noisy.