Scaling Video Understanding via Compact Latent Multi-Agent Collaboration
This work addresses the scalability bottleneck in long-video understanding for multi-modal AI systems, offering a more efficient and information-preserving alternative to rule-based agentic methods.
Multi-modal large language models struggle with long-video tasks due to bounded perception budgets. The proposed MACF framework decouples per-agent budgets from global video complexity via latent communication, achieving state-of-the-art performance on diverse video understanding benchmarks under identical budget constraints.
Multi-modal large language models (MLLMs) advance vision language understanding but face inherent limitations in long-video tasks due to bounded perception context budgets. Existing agentic methods mitigate this via rule-based preprocessing, yet often suffer from information loss, high cost, and reliance on textual intermediates. We propose MACF, an end-to-end Multi-Agent Collaboration Framework that decouples per-agent perception budgets from global video complexity, enabling scalable video understanding while preserving visual fidelity. MACF partitions videos into segments for locally budgeted agents and enables holistic reasoning via an agent-native latent communication protocol. Each agent encodes partial observations into compact, task-sufficient tokens in a shared embedding space, allowing efficient and information-preserving collaboration by a central coordinator. We introduce a curriculum training strategy that progressively enforces semantic alignment, evidence summarization, and cross-agent coordination. Extensive experiments on diverse video understanding benchmarks show that MACF consistently outperforms state-of-the-art MLLMs and multi-agent systems under identical budget constraints, demonstrating the effectiveness of our latent collaboration for scalable video understanding.