Evaluating point-light biological motion in multimodal large language models
This work addresses a key limitation in MLLMs for embodied AI and human-computer interaction, though it is incremental as it introduces a new benchmark rather than a solution.
The paper tackled the problem of evaluating action understanding in multimodal large language models (MLLMs) using point-light displays (PLDs), revealing consistently low performance across models and highlighting gaps in action and spatiotemporal processing.
Humans can extract rich semantic information from minimal visual cues, as demonstrated by point-light displays (PLDs), which consist of sparse sets of dots localized to key joints of the human body. This ability emerges early in development and is largely attributed to human embodied experience. Since PLDs isolate body motion as the sole source of meaning, they represent key stimuli for testing the constraints of action understanding in these systems. Here we introduce ActPLD, the first benchmark to evaluate action processing in MLLMs from human PLDs. Tested models include state-of-the-art proprietary and open-source systems on single-actor and socially interacting PLDs. Our results reveal consistently low performance across models, introducing fundamental gaps in action and spatiotemporal understanding.