Navigating Cultural Chasms: Exploring and Unlocking the Cultural POV of Text-To-Image Models
This work addresses cultural representation issues in AI for users of text-to-image models, but it is incremental as it builds on existing models and evaluation techniques.
The study tackled the problem of understanding cultural biases in text-to-image models by developing an ontology and evaluation methods, resulting in the creation of the CulText2I dataset spanning ten languages and providing insights into cultural encoding.
Text-To-Image (TTI) models, such as DALL-E and StableDiffusion, have demonstrated remarkable prompt-based image generation capabilities. Multilingual encoders may have a substantial impact on the cultural agency of these models, as language is a conduit of culture. In this study, we explore the cultural perception embedded in TTI models by characterizing culture across three hierarchical tiers: cultural dimensions, cultural domains, and cultural concepts. Based on this ontology, we derive prompt templates to unlock the cultural knowledge in TTI models, and propose a comprehensive suite of evaluation techniques, including intrinsic evaluations using the CLIP space, extrinsic evaluations with a Visual-Question-Answer (VQA) model and human assessments, to evaluate the cultural content of TTI-generated images. To bolster our research, we introduce the CulText2I dataset, derived from six diverse TTI models and spanning ten languages. Our experiments provide insights regarding Do, What, Which and How research questions about the nature of cultural encoding in TTI models, paving the way for cross-cultural applications of these models.