PushupBench: Your VLM is not good at counting pushups
Identifies a critical gap in VLM temporal reasoning and provides a benchmark and training signal that transfers to broader video understanding tasks.
VLMs fail at counting repetitions in long videos; PushupBench (446 clips, avg. 36.7s) shows best frontier model achieves 42.1% exact accuracy, while open-source 4B models score ~6%. Fine-tuning on counting with 1k samples improves general video understanding benchmarks (MVBench +2.15, PerceptionTest +1.88, TVBench +4.54).
Large vision-language models (VLMs) can recognize \textit{what} happens in video but fail to count \textit{how many} times. We introduce \textbf{PushupBench}, 446 long-form clips (avg. 36.7s) for evaluating repetition counting. The best frontier model achieves 42.1\% exact accuracy; open-source 4B models score $\sim$6\%, matching supervised baselines. We show that accuracy alone misleads -- weaker models exploit the modal count rather than reason temporally. Fine-tuning on counting with 1k samples transfers to general video understanding: MVBench (+2.15), PerceptionTest (+1.88), TVBench (+4.54), suggesting counting is a proxy for broader temporal reasoning.PushupBench incorporated in \texttt{lmms-eval} (https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/1262) and hosted on (pushupbench.com/)