Can Vision Language Models Understand Mimed Actions?
This work addresses the challenge of improving vision-language models' interpretation of nonverbal communication, specifically mimed actions, which is crucial for more robust human-computer interaction.
The paper tackles the problem of evaluating vision-language models' understanding of mimed actions by introducing the MIME benchmark, which consists of 86 mimed actions with variations, and finds that these models perform significantly worse than humans.
Nonverbal communication (NVC) plays an integral role in human language, but studying NVC in general is challenging because of its broad scope and high variance in interpretation among individuals and cultures. However, mime -- the theatrical technique of suggesting intent using only gesture, expression, and movement -- is a subset of NVC that consists of explicit and embodied actions with much lower human interpretation variance. We argue that a solid understanding of mimed actions is a crucial prerequisite for vision-language models capable of interpreting and commanding more subtle aspects of NVC. Hence, we propose Mime Identification Multimodal Evaluation (MIME), a novel video-based question answering benchmark comprising of 86 mimed actions. Constructed with motion capture data, MIME consists of variations of each action with perturbations applied to the character, background, and viewpoint for evaluating recognition robustness. We find that both open-weight and API-based vision-language models perform significantly worse than humans on MIME, motivating the need for increased research for instilling more robust understanding of human gestures.