Recognizing Co-Speech Gestures in-the-Wild
This work addresses the critical lack of precisely annotated training data for multimodal models to capture semantic co-speech gestures, which is a significant bottleneck for researchers developing more natural human-computer interaction systems.
This paper introduces the Gesture Recognition in the Wild (GRW) dataset, a large-scale benchmark of 156,688 manually annotated video clips, to enable multimodal models to recognize and localize co-speech gestures. The dataset covers a 150-word taxonomy and is used to train video models for gesture classification, word recognition, and temporal localization, establishing new benchmarks for these tasks.
While humans naturally gesture during speech, only a sparse subset of these movements are visually depictive and semantically linked to specific spoken words. Current multimodal models struggle to capture these semantic co-speech gestures, heavily bottlenecked by a lack of precisely annotated training data. To address this, we introduce the Gesture Recognition in the Wild (GRW) dataset, the first large-scale benchmark designed to map unconstrained human gestures to specific words with frame-accurate temporal boundaries. Comprising 156,688 manually annotated video clips, GRW spans a highly diverse 150-word taxonomy of physical actions, spatial descriptors, and abstract concepts. We leverage GRW to train video models to (a) classify gestures as semantic or not, (b) recognize the word corresponding to a co-speech gesture, and (c) temporally localize the gesture. We also use GRW to establish benchmarks for these three tasks.