Harnessing Synthetic Preference Data for Enhancing Temporal Understanding of Video-LLMs
This addresses a key limitation in video understanding for AI applications, though it appears incremental as it builds on existing models with targeted data.
The paper tackles the problem of Video-LLMs underperforming on fine-grained temporal understanding tasks by proposing TimeWarp, a method to create synthetic temporal datasets for fine-tuning, which results in significant absolute performance improvements across seven benchmarks.
While Video Large Language Models (Video-LLMs) have demonstrated remarkable performance across general video understanding benchmarks-particularly in video captioning and descriptive tasks-they consistently underperform on tasks that require fine-grained temporal understanding. This limitation arises due to the lack of visual complexity and temporal nuance in current fine-tuning datasets, leading these models to rely heavily on language-based reasoning rather than truly understanding video dynamics. In this work, we propose TimeWarp, a systematic method to create a targeted synthetic temporal dataset to fine-tune the model's responses to encourage it to focus on the given input video. We introduce a large-scale preference dataset, created using TimeWarp, that captures intricate temporal dynamics often overlooked, grounding the model's responses to visual and temporal information. We demonstrate that when our method is applied to existing models, it significantly improves performance on temporal understanding benchmarks, highlighting the effectiveness of our proposed datasets in advancing temporal understanding in Video-LLMs, resulting in an absolute improvement in performance across seven benchmarks. Code is available at https://github.com/sameepv21/timewarp.