Video Question Answering Using CLIP-Guided Visual-Text Attention
This work addresses video understanding for AI applications, but it is incremental as it builds on existing CLIP and attention methods.
The paper tackles Video Question Answering by proposing a CLIP-guided visual-text attention mechanism to enhance cross-modal learning, achieving state-of-the-art performance on MSVD-QA and MSRVTT-QA datasets.
Cross-modal learning of video and text plays a key role in Video Question Answering (VideoQA). In this paper, we propose a visual-text attention mechanism to utilize the Contrastive Language-Image Pre-training (CLIP) trained on lots of general domain language-image pairs to guide the cross-modal learning for VideoQA. Specifically, we first extract video features using a TimeSformer and text features using a BERT from the target application domain, and utilize CLIP to extract a pair of visual-text features from the general-knowledge domain through the domain-specific learning. We then propose a Cross-domain Learning to extract the attention information between visual and linguistic features across the target domain and general domain. The set of CLIP-guided visual-text features are integrated to predict the answer. The proposed method is evaluated on MSVD-QA and MSRVTT-QA datasets, and outperforms state-of-the-art methods.