Robust Test-time Video-Text Retrieval: Benchmarking and Adapting for Query Shifts
For practitioners deploying video-text retrieval in real-world settings, this work addresses the critical vulnerability to query shifts, offering a robust adaptation method.
The paper introduces a benchmark with 12 video perturbations to evaluate query shifts in video-text retrieval, revealing that shifts amplify hubness. They propose HAT-VTR, a test-time adaptation framework using hubness suppression and temporal consistency, which consistently outperforms prior methods across diverse shifts.
Modern video-text retrieval (VTR) models excel on in-distribution benchmarks but are highly vulnerable to real-world query shifts, where the distribution of query data deviates from the training domain, leading to a sharp performance drop. Existing image-focused robustness solutions are inadequate to handle this vulnerability in video, as they fail to address the complex spatio-temporal dynamics inherent in these shifts. To systematically evaluate this vulnerability, we first introduce a comprehensive benchmark featuring 12 distinct types of video perturbations across five severity degrees. Analysis on this benchmark reveals that query shifts amplify the hubness phenomenon, where a few gallery items become dominant "hubs" that attract a disproportionate number of queries. To mitigate this, we then propose HAT-VTR (Hubness Alleviation for Test-time Video-Text Retrieval), as our baseline test-time adaptation framework designed to directly counteract hubness in VTR. It leverages two key components: a Hubness Suppression Memory to refine similarity scores, and multi-granular losses to enforce temporal feature consistency. Extensive experiments demonstrate that HAT-VTR substantially improves robustness, consistently outperforming prior methods across diverse query shift scenarios, and enhancing model reliability for real-world applications.