VCA: Video Curious Agent for Long Video Understanding
This addresses the challenge of high computational costs in long video analysis for AI applications, though it appears incremental as it builds upon existing VLMs with a novel exploration method.
The paper tackles the problem of long video understanding by introducing VCA, a curiosity-driven video agent that uses self-exploration to navigate video segments, resulting in superior effectiveness and efficiency on multiple benchmarks.
Long video understanding poses unique challenges due to their temporal complexity and low information density. Recent works address this task by sampling numerous frames or incorporating auxiliary tools using LLMs, both of which result in high computational costs. In this work, we introduce a curiosity-driven video agent with self-exploration capability, dubbed as VCA. Built upon VLMs, VCA autonomously navigates video segments and efficiently builds a comprehensive understanding of complex video sequences. Instead of directly sampling frames, VCA employs a tree-search structure to explore video segments and collect frames. Rather than relying on external feedback or reward, VCA leverages VLM's self-generated intrinsic reward to guide its exploration, enabling it to capture the most crucial information for reasoning. Experimental results on multiple long video benchmarks demonstrate our approach's superior effectiveness and efficiency.