Weaver: End-to-End Agentic System Training for Video Interleaved Reasoning
This work addresses video reasoning challenges for AI models, offering a novel agentic system that improves performance on benchmarks, though it appears incremental by building on existing multimodal and reinforcement learning techniques.
The paper tackles the problem of video reasoning by addressing limitations in existing text-centric Chain-of-Thought approaches, such as representational mismatch and limited perceptual acuity, and proposes Weaver, an end-to-end trainable multimodal reasoning agentic system that dynamically invokes tools and uses reinforcement learning, resulting in enhanced performance on complex video reasoning benchmarks, especially for long videos.
Video reasoning constitutes a comprehensive assessment of a model's capabilities, as it demands robust perceptual and interpretive skills, thereby serving as a means to explore the boundaries of model performance. While recent research has leveraged text-centric Chain-of-Thought reasoning to augment these capabilities, such approaches frequently suffer from representational mismatch and restricted by limited perceptual acuity. To address these limitations, we propose Weaver, a novel, end-to-end trainable multimodal reasoning agentic system. Weaver empowers its policy model to dynamically invoke diverse tools throughout the reasoning process, enabling progressive acquisition of crucial visual cues and construction of authentic multimodal reasoning trajectories. Furthermore, we integrate a reinforcement learning algorithm to allow the system to freely explore strategies for employing and combining these tools with trajectory-free data. Extensive experiments demonstrate that our system, Weaver, enhances performance on several complex video reasoning benchmarks, particularly those involving long videos.