EC-Bench: Enumeration and Counting Benchmark for Ultra-Long Videos
This addresses a fundamental challenge for computer vision researchers and practitioners working with long-form video analysis, as it provides a new benchmark to evaluate and improve models for quantitative reasoning in such videos.
The paper tackles the problem of counting and enumerating events in ultra-long videos, which is underexplored in computer vision, by introducing EC-Bench, a benchmark that evaluates these tasks along with temporal evidence grounding, and finds that the best multimodal large language model achieves only 29.98% accuracy on enumeration and 23.74% on counting, far below human performance of 78.57% and 82.97%.
Counting in long videos remains a fundamental yet underexplored challenge in computer vision. Real-world recordings often span tens of minutes or longer and contain sparse, diverse events, making long-range temporal reasoning particularly difficult. However, most existing video counting benchmarks focus on short clips and evaluate only the final numerical answer, providing little insight into what should be counted or whether models consistently identify relevant instances across time. We introduce EC-Bench, a benchmark that jointly evaluates enumeration, counting, and temporal evidence grounding in long-form videos. EC-Bench contains 152 videos longer than 30 minutes and 1,699 queries paired with explicit evidence spans. Across 22 multimodal large language models (MLLMs), the best model achieves only 29.98% accuracy on Enumeration and 23.74% on Counting, while human performance reaches 78.57% and 82.97%, respectively. Our analysis reveals strong relationships between enumeration accuracy, temporal grounding, and counting performance. These results highlight fundamental limitations of current MLLMs and establish EC-Bench as a challenging benchmark for long-form quantitative video reasoning.