FALCONEye: Finding Answers and Localizing Content in ONE-hour-long videos with multi-modal LLMs
This addresses the challenge of efficiently searching hour-long videos for open-ended questions, which is a domain-specific problem for video analysis applications.
The researchers tackled the problem of finding information in hour-long videos by developing FALCONEye, a training-free video agent that uses a multi-modal LLM approach with an exploration-based search algorithm, which outperformed open-source 7B VLMs on their new benchmark and matched or exceeded GPT-4o on certain tasks while reducing inference costs by roughly an order of magnitude.
Finding information in hour-long videos is a challenging task even for top-performing Vision Language Models (VLMs), as encoding visual content quickly exceeds available context windows. To tackle this challenge, we present FALCONEye, a novel video agent based on a training-free, model-agnostic meta-architecture composed of a VLM and a Large Language Model (LLM). FALCONEye answers open-ended questions using an exploration-based search algorithm guided by calibrated confidence from the VLM's answers. We also introduce the FALCON-Bench benchmark, extending Question Answering problem to Video Answer Search-requiring models to return both the answer and its supporting temporal window for open-ended questions in hour-long videos. With just a 7B VLM and a lightweight LLM, FALCONEye outscores all open-source 7B VLMs and comparable agents in FALCON-Bench. It further demonstrates its generalization capability in MLVU benchmark with shorter videos and different tasks, surpassing GPT-4o on single-detail tasks while slashing inference cost by roughly an order of magnitude.