Harnessing Object Grounding for Time-Sensitive Video Understanding
This work addresses a specific bottleneck in video AI for applications requiring precise temporal understanding, representing an incremental improvement over existing methods.
The paper tackles the problem of improving time-sensitive video understanding in video large language models by incorporating grounded object information, proposing GO-Tokenizer as a lightweight module that encodes object data compactly. It shows that this approach outperforms baseline models and textual object descriptions, with gains generalizing across models, datasets, and tasks like reasoning temporal localization and dense captioning.
We propose to improve the time-sensitive video understanding (TSV) capability of video large language models (Video-LLMs) with grounded objects (GO). We hypothesize that TSV tasks can benefit from GO within frames, which is supported by our preliminary experiments on LITA, a state-of-the-art Video-LLM for reasoning temporal localization. While augmenting prompts with textual description of these object annotations improves the performance of LITA, it also introduces extra token length and susceptibility to the noise in object level information. To address this, we propose GO-Tokenizer, a lightweight add-on module for Video-LLMs leveraging off-the-shelf object detectors to encode compact object information on the fly. Experimental results demonstrate that pretraining with GO-Tokenizer outperforms the vanilla Video-LLM and its counterpart utilizing textual description of objects in the prompt. The gain generalizes across different models, datasets and video understanding tasks such as reasoning temporal localization and dense captioning.