BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models
This addresses the need for culturally competent VLMs for global deployment by exposing limitations in existing evaluations, though it is incremental as it builds on prior datasets to create a new benchmark.
The paper tackles the problem of evaluating cultural understanding in vision-language models (VLMs) by introducing BLEnD-Vis, a multimodal benchmark spanning 16 regions with over 21,000 multiple-choice questions and 4,916 images, revealing significant fragility in current VLMs with performance drops under linguistic rephrasing and low cross-modal consistency.
As vision-language models (VLMs) are deployed globally, their ability to understand culturally situated knowledge becomes essential. Yet, existing evaluations largely assess static recall or isolated visual grounding, leaving unanswered whether VLMs possess robust and transferable cultural understanding. We introduce BLEnD-Vis, a multimodal, multicultural benchmark designed to evaluate the robustness of everyday cultural knowledge in VLMs across linguistic rephrasings and visual modalities. Building on the BLEnD dataset, BLEnD-Vis constructs 313 culturally grounded question templates spanning 16 regions and generates three aligned multiple-choice formats: (i) a text-only baseline querying from Region $\to$ Entity, (ii) an inverted text-only variant (Entity $\to$ Region), and (iii) a VQA-style version of (ii) with generated images. The resulting benchmark comprises 4,916 images and over 21,000 multiple-choice question (MCQ) instances, validated through human annotation. BLEnD-Vis reveals significant fragility in current VLM cultural knowledge; models exhibit performance drops under linguistic rephrasing and, whilst visual cues often aid performance, low cross-modal consistency highlights challenges in robustly integrating textual and visual understanding, particularly for lower-resource regions. BLEnD-Vis thus provides a crucial testbed for systematically analysing cultural robustness and multimodal grounding, exposing limitations and guiding the development of more culturally competent VLMs.