Social Caption: Evaluating Social Understanding in Multimodal Models
This work addresses the need for better evaluation metrics for social understanding in AI models, which is crucial for applications in human-computer interaction, but it is incremental as it builds on existing interaction theory and evaluation methods.
The paper tackles the problem of evaluating social understanding in multimodal large language models by introducing the Social Caption framework, which assesses abilities in social inference, holistic analysis, and directed analysis, and analyzes factors like model scale and architecture to provide insights for automated evaluation.
Social understanding abilities are crucial for multimodal large language models (MLLMs) to interpret human social interactions. We introduce Social Caption, a framework grounded in interaction theory to evaluate social understanding abilities of MLLMs along three dimensions: Social Inference (SI), the ability to make accurate inferences about interactions; Holistic Social Analysis (HSA), the ability to generate comprehensive descriptions of interactions; Directed Social Analysis (DSA), the ability to extract relevant social information from interactions. We analyze factors influencing model performance in social understanding, such as scale, architectural design, and spoken context. Experiments with MLLM judges contribute insights about scaling automated evaluation of multimodal social understanding.