Social Genome: Grounded Social Reasoning Abilities of Multimodal Models
This addresses the need for better evaluation of AI systems in interpreting multimodal human interactions, though it is incremental as it focuses on benchmarking rather than novel model development.
The authors tackled the problem of evaluating multimodal AI models' social reasoning abilities by introducing SOCIAL GENOME, the first benchmark for fine-grained, grounded social reasoning, which includes 272 videos and 1,486 annotated reasoning traces with 5,777 steps, and they demonstrated its utility by identifying performance gaps in state-of-the-art models.
Social reasoning abilities are crucial for AI systems to effectively interpret and respond to multimodal human communication and interaction within social contexts. We introduce SOCIAL GENOME, the first benchmark for fine-grained, grounded social reasoning abilities of multimodal models. SOCIAL GENOME contains 272 videos of interactions and 1,486 human-annotated reasoning traces related to inferences about these interactions. These traces contain 5,777 reasoning steps that reference evidence from visual cues, verbal cues, vocal cues, and external knowledge (contextual knowledge external to videos). SOCIAL GENOME is also the first modeling challenge to study external knowledge in social reasoning. SOCIAL GENOME computes metrics to holistically evaluate semantic and structural qualities of model-generated social reasoning traces. We demonstrate the utility of SOCIAL GENOME through experiments with state-of-the-art models, identifying performance gaps and opportunities for future research to improve the grounded social reasoning abilities of multimodal models.