TARA: Simple and Efficient Time Aware Retrieval Adaptation of MLLMs for Video Understanding
This addresses the need for time-aware video retrieval models, offering a versatile solution with incremental improvements in specific domains like action and negation understanding.
The paper tackled the problem of building a time-aware video-text embedding model for retrieval by proposing TARA, a method to adapt Multimodal LLMs without video data, which outperformed existing models on a new chiral benchmark and achieved strong results on standard benchmarks, including state-of-the-art zero-shot performance.
Our objective is to build a general time-aware video-text embedding model for retrieval. To that end, we propose a simple and efficient recipe, dubbed TARA (Time Aware Retrieval Adaptation), to adapt Multimodal LLMs (MLLMs) to a time-aware video-text embedding model without using any video data at all. For evaluating time-awareness in retrieval, we propose a new benchmark with temporally opposite (chiral) actions as hard negatives and curated splits for chiral and non-chiral actions. We show that TARA outperforms all existing video-text models on this chiral benchmark while also achieving strong results on standard benchmarks. Furthermore, we discover additional benefits of TARA beyond time-awareness: (i) TARA embeddings are negation-aware as shown in NegBench benchmark that evaluates negation in video retrieval, (ii) TARA achieves state of the art performance on verb and adverb understanding in videos. Overall, TARA yields a strong, versatile, time-aware video-text embedding model with state of the art zero-shot performance.