NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training
This work addresses the challenge of efficient pre-training for vision-language models by mitigating noise in web data, which is crucial for reducing computational costs and improving performance in AI applications.
The paper tackles the problem of noisy and incomplete web data in vision-language pre-training by proposing NEVLP, a noise-robust framework that uses noise-adaptive and concept-enhanced learning strategies, achieving state-of-the-art performance with less pre-training data across tasks like image-text retrieval, image captioning, and visual question answering.
The success of Vision Language Models (VLMs) on various vision-language tasks heavily relies on pre-training with large scale web-crawled datasets. However, the noisy and incomplete nature of web data makes dataset scale crucial for performance, rendering end-to-end training increasingly prohibitive. In this paper, we propose NEVLP, a noise-robust framework for efficient vision-language pre-training that requires less pre-training data. Specifically, we bridge the modality gap between a frozen image encoder and a large language model with a transformer and introduce two innovative learning strategies: noise-adaptive learning and concept-enhanced learning to mitigate the impact of noise. In noise-adaptive learning, we estimate the noise probability of each image-text pair based on the transformer's memorization effect and employ noise-adaptive regularization on image-text contrastive learning to condition cross-modal alignment. In concept-enhanced learning, we enrich incomplete text by incorporating visual concepts (objects in the image) to provide prior information about existing objects for image-text matching and image-grounded text generation, thereby mitigating text incompletion. Our framework effectively utilizes noisy web data and achieves state-of-the-art performance with less pre-training data across a wide range of vision-language tasks, including image-text retrieval, image captioning, and visual question answering.