RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understanding
This addresses the need for accurate visual culture understanding in VLMs, which is critical as they become integrated into daily life, though it is incremental as it extends prior RAG work to multimodal scenarios.
The paper tackles the problem of vision-language models falling short in interpreting cultural nuances by introducing RAVENEA, a benchmark for multimodal retrieval-augmented visual culture understanding, showing that lightweight VLMs with culture-aware retrieval outperform non-augmented counterparts by at least 3.2% on cVQA and 6.2% on cIC.
As vision-language models (VLMs) become increasingly integrated into daily life, the need for accurate visual culture understanding is becoming critical. Yet, these models frequently fall short in interpreting cultural nuances effectively. Prior work has demonstrated the effectiveness of retrieval-augmented generation (RAG) in enhancing cultural understanding in text-only settings, while its application in multimodal scenarios remains underexplored. To bridge this gap, we introduce RAVENEA (Retrieval-Augmented Visual culturE uNdErstAnding), a new benchmark designed to advance visual culture understanding through retrieval, focusing on two tasks: culture-focused visual question answering (cVQA) and culture-informed image captioning (cIC). RAVENEA extends existing datasets by integrating over 10,000 Wikipedia documents curated and ranked by human annotators. With RAVENEA, we train and evaluate seven multimodal retrievers for each image query, and measure the downstream impact of retrieval-augmented inputs across fourteen state-of-the-art VLMs. Our results show that lightweight VLMs, when augmented with culture-aware retrieval, outperform their non-augmented counterparts (by at least 3.2% absolute on cVQA and 6.2% absolute on cIC). This highlights the value of retrieval-augmented methods and culturally inclusive benchmarks for multimodal understanding.