Contrastive Visual-Linguistic Pretraining
This work addresses issues in multi-modality pretraining for vision-language tasks, offering an incremental improvement over prior methods.
The paper tackles the domain gap and noisy label problems in existing multi-modality representation learning methods like ViLBERT and LXMERT by proposing CVLP, which uses a visual self-supervised contrastive loss, and shows superior performance on tasks such as VQA, GQA, and NLVR2.
Several multi-modality representation learning approaches such as LXMERT and ViLBERT have been proposed recently. Such approaches can achieve superior performance due to the high-level semantic information captured during large-scale multimodal pretraining. However, as ViLBERT and LXMERT adopt visual region regression and classification loss, they often suffer from domain gap and noisy label problems, based on the visual features having been pretrained on the Visual Genome dataset. To overcome these issues, we propose unbiased Contrastive Visual-Linguistic Pretraining (CVLP), which constructs a visual self-supervised loss built upon contrastive learning. We evaluate CVLP on several down-stream tasks, including VQA, GQA and NLVR2 to validate the superiority of contrastive learning on multi-modality representation learning. Our code is available at: https://github.com/ArcherYunDong/CVLP-.