Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension
This addresses the challenge of long video understanding for users of video-language models, offering a cost-effective and compatible solution, though it is incremental as it builds on existing retrieval-augmented generation methods.
The paper tackles the problem of limited context in large video-language models for long video comprehension by proposing Video-RAG, a training-free pipeline that uses visually-aligned auxiliary texts, resulting in significant performance gains on benchmarks like Video-MME, MLVU, and LongVideoBench, including superior performance over proprietary models like Gemini-1.5-Pro and GPT-4o with a 72B model.
Existing large video-language models (LVLMs) struggle to comprehend long videos correctly due to limited context. To address this problem, fine-tuning long-context LVLMs and employing GPT-based agents have emerged as promising solutions. However, fine-tuning LVLMs would require extensive high-quality data and substantial GPU resources, while GPT-based agents would rely on proprietary models (e.g., GPT-4o). In this paper, we propose Video Retrieval-Augmented Generation (Video-RAG), a training-free and cost-effective pipeline that employs visually-aligned auxiliary texts to help facilitate cross-modality alignment while providing additional information beyond the visual content. Specifically, we leverage open-source external tools to extract visually-aligned information from pure video data (e.g., audio, optical character, and object detection), and incorporate the extracted information into an existing LVLM as auxiliary texts, alongside video frames and queries, in a plug-and-play manner. Our Video-RAG offers several key advantages: (i) lightweight with low computing overhead due to single-turn retrieval; (ii) easy implementation and compatibility with any LVLM; and (iii) significant, consistent performance gains across long video understanding benchmarks, including Video-MME, MLVU, and LongVideoBench. Notably, our model demonstrates superior performance over proprietary models like Gemini-1.5-Pro and GPT-4o when utilized with a 72B model.