RAVEN: An Agentic Framework for Multimodal Entity Discovery from Large-Scale Video Collections
This addresses the challenge of scalable information retrieval from video data for users in domains requiring efficient content analysis, though it appears incremental as it builds on existing VLMs and LLMs with a flexible, model-agnostic approach.
The paper tackles the problem of multimodal entity discovery and retrieval from large-scale video collections by introducing RAVEN, an adaptive AI agent framework that synthesizes visual, audio, and textual data to produce structured representations, enabling applications like personalized search and content discovery.
We present RAVEN an adaptive AI agent framework designed for multimodal entity discovery and retrieval in large-scale video collections. Synthesizing information across visual, audio, and textual modalities, RAVEN autonomously processes video data to produce structured, actionable representations for downstream tasks. Key contributions include (1) a category understanding step to infer video themes and general-purpose entities, (2) a schema generation mechanism that dynamically defines domain-specific entities and attributes, and (3) a rich entity extraction process that leverages semantic retrieval and schema-guided prompting. RAVEN is designed to be model-agnostic, allowing the integration of different vision-language models (VLMs) and large language models (LLMs) based on application-specific requirements. This flexibility supports diverse applications in personalized search, content discovery, and scalable information retrieval, enabling practical applications across vast datasets.