Learning Video Representations using Contrastive Bidirectional Transformer
This work addresses the problem of learning effective video features for tasks such as classification and captioning, representing an incremental improvement over existing methods.
The paper tackles video representation learning by proposing a self-supervised approach that extends BERT to sequences of feature vectors using noise contrastive estimation, resulting in significantly improved performance on downstream tasks like video classification, captioning, and segmentation.
This paper proposes a self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods. Our method extends the BERT model for text sequences to the case of sequences of real-valued feature vectors, by replacing the softmax loss with noise contrastive estimation (NCE). We also show how to learn representations from sequences of visual features and sequences of words derived from ASR (automatic speech recognition), and show that such cross-modal training (when possible) helps even more.