Leveraging Generative Language Models for Weakly Supervised Sentence Component Analysis in Video-Language Joint Learning
This work addresses the limitation of textual understanding in video-language tasks for AI researchers, but it is incremental as it builds on existing methods with a novel application.
The paper tackles the problem of poor textual comprehension in video-language models by using a pre-trained LLM to generate text samples targeting sentence components and estimating their importance, resulting in relative improvements of up to 8.3% in video-text retrieval and up to 13.7% in video moment retrieval.
A thorough comprehension of textual data is a fundamental element in multi-modal video analysis tasks. However, recent works have shown that the current models do not achieve a comprehensive understanding of the textual data during the training for the target downstream tasks. Orthogonal to the previous approaches to this limitation, we postulate that understanding the significance of the sentence components according to the target task can potentially enhance the performance of the models. Hence, we utilize the knowledge of a pre-trained large language model (LLM) to generate text samples from the original ones, targeting specific sentence components. We propose a weakly supervised importance estimation module to compute the relative importance of the components and utilize them to improve different video-language tasks. Through rigorous quantitative analysis, our proposed method exhibits significant improvement across several video-language tasks. In particular, our approach notably enhances video-text retrieval by a relative improvement of 8.3\% in video-to-text and 1.4\% in text-to-video retrieval over the baselines, in terms of R@1. Additionally, in video moment retrieval, average mAP shows a relative improvement ranging from 2.0\% to 13.7 \% across different baselines.