HOP: Heterogeneous Topology-based Multimodal Entanglement for Co-Speech Gesture Generation
This addresses the challenge of generating realistic gestures for human-computer interaction or virtual agents, though it appears incremental as it builds on existing multimodal methods.
The paper tackles the problem of generating diverse and coherent co-speech gestures by proposing HOP, a method that models heterogeneous entanglement between gesture motion, audio rhythm, and text semantics, achieving state-of-the-art performance with more natural and expressive results.
Co-speech gestures are crucial non-verbal cues that enhance speech clarity and expressiveness in human communication, which have attracted increasing attention in multimodal research. While the existing methods have made strides in gesture accuracy, challenges remain in generating diverse and coherent gestures, as most approaches assume independence among multimodal inputs and lack explicit modeling of their interactions. In this work, we propose a novel multimodal learning method named HOP for co-speech gesture generation that captures the heterogeneous entanglement between gesture motion, audio rhythm, and text semantics, enabling the generation of coordinated gestures. By leveraging spatiotemporal graph modeling, we achieve the alignment of audio and action. Moreover, to enhance modality coherence, we build the audio-text semantic representation based on a reprogramming module, which is beneficial for cross-modality adaptation. Our approach enables the trimodal system to learn each other's features and represent them in the form of topological entanglement. Extensive experiments demonstrate that HOP achieves state-of-the-art performance, offering more natural and expressive co-speech gesture generation. More information, codes, and demos are available here: https://star-uu-wang.github.io/HOP/