Retrieval-based Knowledge Augmented Vision Language Pre-training
This work addresses the problem of inefficient knowledge utilization in vision-language models for researchers and practitioners, representing a novel method rather than an incremental improvement.
The paper tackles the challenge of effectively integrating world knowledge into vision-language pre-training by proposing REAVL, a retrieval-based framework that adaptively identifies and incorporates informative knowledge, achieving state-of-the-art performance on knowledge-based tasks and competitive results on general tasks with only 0.2% of the pre-training data.
With the recent progress in large-scale vision and language representation learning, Vision Language Pre-training (VLP) models have achieved promising improvements on various multi-modal downstream tasks. Albeit powerful, these models have not fully leveraged world knowledge to their advantage. A key challenge of knowledge-augmented VLP is the lack of clear connections between knowledge and multi-modal data. Moreover, not all knowledge present in images/texts is useful, therefore prior approaches often struggle to effectively integrate knowledge, visual, and textual information. In this study, we propose REtrieval-based knowledge Augmented Vision Language (REAVL), a novel knowledge-augmented pre-training framework to address the above issues. For the first time, we introduce a knowledge-aware self-supervised learning scheme that efficiently establishes the correspondence between knowledge and multi-modal data and identifies informative knowledge to improve the modeling of alignment and interactions between visual and textual modalities. By adaptively integrating informative knowledge with visual and textual information, REAVL achieves new state-of-the-art performance uniformly on knowledge-based vision-language understanding and multi-modal entity linking tasks, as well as competitive results on general vision-language tasks while only using 0.2% pre-training data of the best models. Our model shows strong sample efficiency and effective knowledge utilization.