Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialects
This addresses the problem of evaluating multilingual vision-language models for culturally grounded understanding in underrepresented contexts like Bengali culture, though it is incremental as it builds on existing benchmark methodologies.
The authors tackled the underrepresentation of Bangla culture in multimodal evaluation by introducing BanglaVerse, a benchmark with 1,152 images across nine domains, expanded to four languages and five Bangla dialects, yielding ~32.3K artifacts. Their experiments showed that evaluating only standard Bangla overestimates model capability, with performance dropping under dialectal variation, especially for caption generation, and historically linked languages like Hindi and Urdu retaining some cultural meaning but being weaker for structured reasoning.
Bangla culture is richly expressed through region, dialect, history, food, politics, media, and everyday visual life, yet it remains underrepresented in multimodal evaluation. To address this gap, we introduce BanglaVerse, a culturally grounded benchmark for evaluating multilingual vision-language models (VLMs) on Bengali culture across historically linked languages and regional dialects. Built from 1,152 manually curated images across nine domains, the benchmark supports visual question answering and captioning, and is expanded into four languages and five Bangla dialects, yielding ~32.3K artifacts. Our experiments show that evaluating only standard Bangla overestimates true model capability: performance drops under dialectal variation, especially for caption generation, while historically linked languages such as Hindi and Urdu retain some cultural meaning but remain weaker for structured reasoning. Across domains, the main bottleneck is missing cultural knowledge rather than visual grounding alone, with knowledge-intensive categories. These findings position BanglaVerse as a more realistic test bed for measuring culturally grounded multimodal understanding under linguistic variation.