SocialGesture: Delving into Multi-person Gesture Understanding
This addresses the problem of understanding social context in gestures for researchers in computer vision and human-computer interaction, though it is incremental as it focuses on dataset creation.
The authors tackled the lack of multi-person gesture datasets by introducing SocialGesture, a large-scale dataset for multi-person gesture analysis, and found that current gesture recognition models have limitations, as highlighted through a novel VQA task.
Previous research in human gesture recognition has largely overlooked multi-person interactions, which are crucial for understanding the social context of naturally occurring gestures. This limitation in existing datasets presents a significant challenge in aligning human gestures with other modalities like language and speech. To address this issue, we introduce SocialGesture, the first large-scale dataset specifically designed for multi-person gesture analysis. SocialGesture features a diverse range of natural scenarios and supports multiple gesture analysis tasks, including video-based recognition and temporal localization, providing a valuable resource for advancing the study of gesture during complex social interactions. Furthermore, we propose a novel visual question answering (VQA) task to benchmark vision language models'(VLMs) performance on social gesture understanding. Our findings highlight several limitations of current gesture recognition models, offering insights into future directions for improvement in this field. SocialGesture is available at huggingface.co/datasets/IrohXu/SocialGesture.