VLR-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generation
This provides a new benchmark for researchers to test vision-language models on retrieval-augmented generation, but it is incremental as it builds on existing evaluation datasets.
The authors tackled the problem of evaluating vision-language models for retrieval-augmented generation by proposing VLR-Bench, a multilingual benchmark dataset with 32,000 automatically generated examples, and verified its performance using a state-of-the-art model.
We propose the VLR-Bench, a visual question answering (VQA) benchmark for evaluating vision language models (VLMs) based on retrieval augmented generation (RAG). Unlike existing evaluation datasets for external knowledge-based VQA, the proposed VLR-Bench includes five input passages. This allows testing of the ability to determine which passage is useful for answering a given query, a capability lacking in previous research. In this context, we constructed a dataset of 32,000 automatically generated instruction-following examples, which we denote as VLR-IF. This dataset is specifically designed to enhance the RAG capabilities of VLMs by enabling them to learn how to generate appropriate answers based on input passages. We evaluated the validity of the proposed benchmark and training data and verified its performance using the state-of-the-art Llama3-based VLM, the Llava-Llama-3 model. The proposed VLR-Bench and VLR-IF datasets are publicly available online.