Bodily Behaviors in Social Interaction: Novel Annotations and State-of-the-Art Evaluation
This work addresses the challenge of understanding social behavior for AI systems, but it is incremental as it builds on existing methods and datasets.
The authors tackled the problem of automatically detecting subtle bodily behaviors like gesturing and grooming in social interactions by introducing BBSI, a novel annotation set of 26 hours of group interactions with 15 body language classes, and adapted the Pyramid Dilated Attention Network for detection, achieving promising results with room for improvement.
Body language is an eye-catching social signal and its automatic analysis can significantly advance artificial intelligence systems to understand and actively participate in social interactions. While computer vision has made impressive progress in low-level tasks like head and body pose estimation, the detection of more subtle behaviors such as gesturing, grooming, or fumbling is not well explored. In this paper we present BBSI, the first set of annotations of complex Bodily Behaviors embedded in continuous Social Interactions in a group setting. Based on previous work in psychology, we manually annotated 26 hours of spontaneous human behavior in the MPIIGroupInteraction dataset with 15 distinct body language classes. We present comprehensive descriptive statistics on the resulting dataset as well as results of annotation quality evaluations. For automatic detection of these behaviors, we adapt the Pyramid Dilated Attention Network (PDAN), a state-of-the-art approach for human action detection. We perform experiments using four variants of spatial-temporal features as input to PDAN: Two-Stream Inflated 3D CNN, Temporal Segment Networks, Temporal Shift Module and Swin Transformer. Results are promising and indicate a great room for improvement in this difficult task. Representing a key piece in the puzzle towards automatic understanding of social behavior, BBSI is fully available to the research community.