Toward Socially Aware Vision-Language Models: Evaluating Cultural Competence Through Multimodal Story Generation
This addresses the need for responsible AI systems in diverse cultural contexts, though it is incremental as it builds on prior evaluations of cultural awareness in text-only models and VLM object recognition tasks.
The paper tackled the problem of evaluating cultural competence in Vision-Language Models (VLMs) through multimodal story generation, revealing significant cultural adaptation capabilities with rich culturally-specific vocabulary but also concerning limitations like dramatic variation across architectures and inverse cultural alignment.
As Vision-Language Models (VLMs) achieve widespread deployment across diverse cultural contexts, ensuring their cultural competence becomes critical for responsible AI systems. While prior work has evaluated cultural awareness in text-only models and VLM object recognition tasks, no research has systematically assessed how VLMs adapt outputs when cultural identity cues are embedded in both textual prompts and visual inputs during generative tasks. We present the first comprehensive evaluation of VLM cultural competence through multimodal story generation, developing a novel multimodal framework that perturbs cultural identity and evaluates 5 contemporary VLMs on a downstream task: story generation. Our analysis reveals significant cultural adaptation capabilities, with rich culturally-specific vocabulary spanning names, familial terms, and geographic markers. However, we uncover concerning limitations: cultural competence varies dramatically across architectures, some models exhibit inverse cultural alignment, and automated metrics show architectural bias contradicting human assessments. Cross-modal evaluation shows that culturally distinct outputs are indeed detectable through visual-semantic similarity (28.7% within-nationality vs. 0.2% cross-nationality recall), yet visual-cultural understanding remains limited. In essence, we establish the promise and challenges of cultural competence in multimodal AI. We publicly release our codebase and data: https://github.com/ArkaMukherjee0/mmCultural