MAVIS: A Benchmark for Multimodal Source Attribution in Long-form Visual Question Answering
This work addresses the need for reliable AI-generated answers with citations in multimodal contexts, though it is incremental as it builds on existing RAG methods.
The authors tackled the problem of multimodal source attribution in long-form visual question answering by introducing MAVIS, a benchmark with 157K visual QA instances and fine-grained automatic metrics, finding that multimodal RAG improves informativeness and fluency but weakens groundedness for images compared to text.
Source attribution aims to enhance the reliability of AI-generated answers by including references for each statement, helping users validate the provided answers. However, existing work has primarily focused on text-only scenario and largely overlooked the role of multimodality. We introduce MAVIS, the first benchmark designed to evaluate multimodal source attribution systems that understand user intent behind visual questions, retrieve multimodal evidence, and generate long-form answers with citations. Our dataset comprises 157K visual QA instances, where each answer is annotated with fact-level citations referring to multimodal documents. We develop fine-grained automatic metrics along three dimensions of informativeness, groundedness, and fluency, and demonstrate their strong correlation with human judgments. Our key findings are threefold: (1) LVLMs with multimodal RAG generate more informative and fluent answers than unimodal RAG, but they exhibit weaker groundedness for image documents than for text documents, a gap amplified in multimodal settings. (2) Given the same multimodal documents, there is a trade-off between informativeness and groundedness across different prompting methods. (3) Our proposed method highlights mitigating contextual bias in interpreting image documents as a crucial direction for future research. The dataset and experimental code are available at https://github.com/seokwon99/MAVIS