QSVideo: Query-Conditioned Semantic Temporal Retrieval for Video Understanding
For researchers working on video understanding with VLMs, QSVideo provides a practical retrieval framework to handle long videos, though it is an incremental improvement over existing retrieval methods.
QSVideo improves video understanding in VLMs under strict frame limits by addressing biased relevance, limited diversity, and temporal collapse in retrieval. It achieves significant gains on long and streaming video benchmarks, e.g., +5.2% on EgoSchema and +4.1% on Next-QA.
The performance of vision-language models (VLMs) in video understanding declines with increasing video duration, as video moments unrelated to the query confuse their language components. Multimodal retrieval has emerged as a critical component of video understanding, addressing this challenge by localizing key visual evidence. However, existing multimodal retrieval methods suffer from biased relevance estimation, limited diversity, and temporal collapse. In this paper, we propose QSVideo, a unified framework that systematically addresses relevance, diversity, and temporal modeling in video retrieval. We first introduce a query-conditioned semantic ranker, QSRanker, which reformulates arbitrary questions into retrieval-friendly queries and estimates structured relevance along object, action, and location dimensions. Building upon this, we design QSRetrieval to jointly optimize relevance and diversity for more informative frame selection. Moreover, we propose temporal alignment strategies tailored for both long and streaming videos to improve evidence recall. Extensive experiments on long and streaming video benchmarks demonstrate that QSVideo greatly enhances video VLM performance under strict frame limit constraints. The code is available at https://github.com/human-analysis/QSVideo.