Contrastive Language-Image Pre-Training with Knowledge Graphs
This addresses the need for better semantic understanding in vision-language models for AI applications, though it appears incremental as it builds directly on CLIP.
The paper tackles the problem of limited semantic connections in existing vision-language pre-training by proposing Knowledge-CLIP, which injects knowledge graphs into CLIP to improve semantic alignment and reasoning. Experiments show it outperforms CLIP and baselines on various downstream tasks.
Recent years have witnessed the fast development of large-scale pre-training frameworks that can extract multi-modal representations in a unified form and achieve promising performances when transferred to downstream tasks. Nevertheless, existing approaches mainly focus on pre-training with simple image-text pairs, while neglecting the semantic connections between concepts from different modalities. In this paper, we propose a knowledge-based pre-training framework, dubbed Knowledge-CLIP, which injects semantic information into the widely used CLIP model. Through introducing knowledge-based objectives in the pre-training process and utilizing different types of knowledge graphs as training data, our model can semantically align the representations in vision and language with higher quality, and enhance the reasoning ability across scenarios and modalities. Extensive experiments on various vision-language downstream tasks demonstrate the effectiveness of Knowledge-CLIP compared with the original CLIP and competitive baselines.