Beyond Screenshots: Evaluating VLMs' Understanding of UI Animations
For AI agents operating on user interfaces, this work identifies limitations in current VLMs' ability to understand dynamic UI animations, highlighting a gap in existing static-focused evaluations.
The paper introduces AniMINT, a dataset of 300 annotated UI animation videos, and evaluates VLMs on understanding UI animations. Results show VLMs reliably detect primitive motion but struggle with high-level interpretation, with performance gaps relative to humans.
AI agents operating on user interfaces must understand how interfaces communicate state and feedback to act reliably. As a core communicative modality, animations are increasingly used in modern interfaces, serving critical functional purposes beyond mere aesthetics. Thus, understanding UI animation is essential for comprehensive interface interpretation. However, recent studies of Vision Language Models (VLMs) for UI understanding have focused primarily on static screenshots, leaving it unclear how well these models handle dynamic UI animations. To address this gap, we created AniMINT, a novel dataset of 300 densely annotated UI animation videos. We systematically evaluate state-of-the-art VLMs on UI animation understanding, including their abilities to perceive the animation effects, identify animation purposes, and interpret animation meaning. Our results show that VLMs can reliably detect primitive motion. However, their high-level animation interpretation remains inconsistent, with substantial gaps relative to human performance. Finally, we use Motion, Context, and Perceptual Cues (MCPC) to probe factors affecting VLM performance, revealing key bottlenecks and directions for future improvement.