I see what you mean: Co-Speech Gestures for Reference Resolution in Multimodal Dialogue
This work addresses the understudied computational problem of gesture-based reference resolution for advancing naturalistic human-machine interaction, though it is incremental in scope.
The paper tackled the problem of how representational co-speech gestures refer to objects in multimodal dialogue by introducing a reference resolution task and a self-supervised pre-training approach for gesture embeddings. The result showed that the learned embeddings aligned with expert annotations and improved reference resolution accuracy, especially when using multimodal representations and dialogue history.
In face-to-face interaction, we use multiple modalities, including speech and gestures, to communicate information and resolve references to objects. However, how representational co-speech gestures refer to objects remains understudied from a computational perspective. In this work, we address this gap by introducing a multimodal reference resolution task centred on representational gestures, while simultaneously tackling the challenge of learning robust gesture embeddings. We propose a self-supervised pre-training approach to gesture representation learning that grounds body movements in spoken language. Our experiments show that the learned embeddings align with expert annotations and have significant predictive power. Moreover, reference resolution accuracy further improves when (1) using multimodal gesture representations, even when speech is unavailable at inference time, and (2) leveraging dialogue history. Overall, our findings highlight the complementary roles of gesture and speech in reference resolution, offering a step towards more naturalistic models of human-machine interaction.