CAViAR: Critic-Augmented Video Agentic Reasoning
This addresses the challenge of leveraging existing perception capabilities for more complex video reasoning tasks, which is incremental as it builds on prior work like Visual Programming and ViperGPT.
The paper tackles the problem of complex video reasoning by developing a large language model agent that uses video modules as tools and a critic to evaluate sequences, achieving strong performance on benchmarks like LVBench, Neptune, and ActivityNet-RTL.
Video understanding has seen significant progress in recent years, with models' performance on perception from short clips continuing to rise. Yet, multiple recent benchmarks, such as LVBench, Neptune, and ActivityNet-RTL, show performance wanes for tasks requiring complex reasoning on videos as queries grow more complex and videos grow longer. In this work, we ask: can existing perception capabilities be leveraged to successfully perform more complex video reasoning? In particular, we develop a large language model agent given access to video modules as subagents or tools. Rather than following a fixed procedure to solve queries as in previous work such as Visual Programming, ViperGPT, and MoReVQA, the agent uses the results of each call to a module to determine subsequent steps. Inspired by work in the textual reasoning domain, we introduce a critic to distinguish between instances of successful and unsuccessful sequences from the agent. We show that the combination of our agent and critic achieve strong performance on the previously-mentioned datasets.