Human Semantic Representations of Social Interactions from Moving Shapes
This addresses the problem of understanding human social perception for cognitive science and AI, but it is incremental as it builds on prior work on visual features.
The study investigated how humans perceive social interactions from moving shapes, finding that semantic models, particularly verb-based embeddings from descriptions, best account for human similarity judgments and complement visual features.
Humans are social creatures who readily recognize various social interactions from simple display of moving shapes. While previous research has often focused on visual features, we examine what semantic representations that humans employ to complement visual features. In Study 1, we directly asked human participants to label the animations based on their impression of moving shapes. We found that human responses were distributed. In Study 2, we measured the representational geometry of 27 social interactions through human similarity judgments and compared it with model predictions based on visual features, labels, and semantic embeddings from animation descriptions. We found that semantic models provided complementary information to visual features in explaining human judgments. Among the semantic models, verb-based embeddings extracted from descriptions account for human similarity judgments the best. These results suggest that social perception in simple displays reflects the semantic structure of social interactions, bridging visual and abstract representations.