ActBERT: Learning Global-Local Video-Text Representations
This work addresses the challenge of video-text representation learning for applications in retrieval, captioning, and question answering, representing an incremental improvement over existing methods.
The paper tackles the problem of learning joint video-text representations from unlabeled data by introducing ActBERT, which leverages global action information and local regional objects to model detailed visual-text relations, and it significantly outperforms state-of-the-art methods on tasks like text-video retrieval and video captioning.
In this paper, we introduce ActBERT for self-supervised learning of joint video-text representations from unlabeled data. First, we leverage global action information to catalyze the mutual interactions between linguistic texts and local regional objects. It uncovers global and local visual clues from paired video sequences and text descriptions for detailed visual and text relation modeling. Second, we introduce an ENtangled Transformer block (ENT) to encode three sources of information, i.e., global actions, local regional objects, and linguistic descriptions. Global-local correspondences are discovered via judicious clues extraction from contextual information. It enforces the joint videotext representation to be aware of fine-grained objects as well as global human intention. We validate the generalization capability of ActBERT on downstream video-and language tasks, i.e., text-video clip retrieval, video captioning, video question answering, action segmentation, and action step localization. ActBERT significantly outperforms the state-of-the-arts, demonstrating its superiority in video-text representation learning.