CVLUE: A New Benchmark Dataset for Chinese Vision-Language Understanding Evaluation
This addresses the problem of cultural bias in evaluating Chinese vision-language models, providing a more suitable benchmark for researchers and developers in this domain, though it is incremental as it builds on existing dataset creation methods.
The authors tackled the cultural bias in existing Chinese vision-language datasets by creating CVLUE, a new benchmark dataset with images representative of Chinese culture, and found that fine-tuning on Chinese culture-related datasets enhances model understanding of Chinese culture.
Despite the rapid development of Chinese vision-language models (VLMs), most existing Chinese vision-language (VL) datasets are constructed on Western-centric images from existing English VL datasets. The cultural bias in the images makes these datasets unsuitable for evaluating VLMs in Chinese culture. To remedy this issue, we present a new Chinese Vision- Language Understanding Evaluation (CVLUE) benchmark dataset, where the selection of object categories and images is entirely driven by Chinese native speakers, ensuring that the source images are representative of Chinese culture. The benchmark contains four distinct VL tasks ranging from image-text retrieval to visual question answering, visual grounding and visual dialogue. We present a detailed statistical analysis of CVLUE and provide a baseline performance analysis with several open-source multilingual VLMs on CVLUE and its English counterparts to reveal their performance gap between English and Chinese. Our in-depth category-level analysis reveals a lack of Chinese cultural knowledge in existing VLMs. We also find that fine-tuning on Chinese culture-related VL datasets effectively enhances VLMs' understanding of Chinese culture.