CultureCLIP: Empowering CLIP with Cultural Awareness through Synthetic Images and Contextualized Captions
This addresses the issue of cultural bias in AI for users in diverse cultural contexts, but it is incremental as it builds on existing CLIP methods.
The paper tackled the problem of pretrained vision-language models like CLIP struggling with nuanced cultural cues by fine-tuning CLIP on a synthetic cultural dataset, resulting in up to a 5.49% improvement in fine-grained concept recognition on culture-specific benchmarks.
Pretrained vision-language models (VLMs) such as CLIP excel in general multimodal comprehension but often struggle to capture nuanced, context-dependent visual cues. This makes it difficult to distinguish between similar-looking concepts with potentially different cultural meanings. Such deficiencies are mainly due to a limited amount of high-quality cultural data, contextual information, and the lack of negative examples that highlight subtle differences. To mitigate this, we design a data curation pipeline leveraging open-sourced VLMs and text-to-image models to construct CulTwin, a synthetic cultural dataset. This dataset consists of paired concept-caption-image triplets, where concepts visually resemble each other but are culturally different. Then, we fine-tune CLIP on CulTwin to develop CultureCLIP, which aligns cultural concepts with contextually enhanced captions and synthetic images through tailored contrastive learning. Experiments on culture-specific benchmarks show that CultureCLIP outperforms the base CLIP, achieving up to a notable 5.49% improvement in fine-grained concept recognition on certain tasks while preserving CLIP's original generalization ability, validating the effectiveness of our data synthesis and VLM backbone training paradigm in capturing subtle cultural distinctions.