VidNum-1.4K: A Comprehensive Benchmark for Video-based Numerical Reasoning
This addresses the need for a comprehensive diagnostic testbed to assess multi-step numerical logic in VLMs for video-based AI applications, though it is incremental as it builds on existing benchmarking efforts.
The authors tackled the problem of evaluating genuine numerical reasoning in Vision-Language Models (VLMs) by introducing VidNum-1.4K, a benchmark with 1,379 video-question pairs, and found that state-of-the-art VLMs like Gemini-3.1-pro achieve only 60% accuracy, revealing a significant reasoning gap.
Video-based numerical reasoning provides a premier arena for testing whether Vision-Language Models (VLMs) truly "understand" real-world dynamics, as accurate numerical deduction necessitates a profound grasp of temporal events, object permanence, and compositional logic beyond superficial pattern matching. However, existing benchmarks are often confined to narrow domains, such as repetitive athletic motions, or treat simple counting merely as a superficial regression task, failing to assess multi-step numerical logic within the inherent complexity of real-world multimedia content. We introduce VidNum-1.4K, a comprehensive VideoQA benchmark comprising 1,379 strictly human-annotated video-question pairs designed to evaluate genuine numerical reasoning across highly diverse environments, encompassing object, action, and event quantification. The VidNum-1.4K is uniquely structured into a three-level hierarchy that evolves from direct visual perception to video-based compositional numerical reasoning, requiring models to perform arithmetic operations, comparisons, and logical deductions grounded in temporal evidence. Our evaluations across a diverse suite of state-of-the-art VLMs reveal a striking reasoning gap: while the Gemini-3.1-pro barely reaches a 60% accuracy threshold, representative open-source families struggle heavily in the 25%--45% range. These findings demonstrate that current VLMs still lack a stable "internal world model", positioning VidNum-1.4K as a demanding diagnostic testbed for the next generation of numerical video intelligence.