Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG
This work is significant for researchers and practitioners in RAG, as it re-evaluates the true sources of progress in multilingual and visually rich RAG systems by demonstrating that better document representation is key.
This paper investigates the performance gaps in Retrieval-Augmented Generation (RAG) benchmarks for multilingual and visually rich documents. It finds that improved document representation, rather than novel retrieval mechanisms, is the primary driver of performance gains, demonstrating that BM25 can recover large gaps by systematically varying transcription and preprocessing methods.
Retrieval-augmented generation (RAG) is a common way to ground language models in external documents and up-to-date information. Classical retrieval systems relied on lexical methods such as BM25, which rank documents by term overlap with corpus-level weighting. End-to-end multimodal retrievers trained on large query-document datasets claim substantial improvements over these approaches, especially for multilingual documents with complex visual layouts. We demonstrate that better document representation is the primary driver of benchmark improvements. By systematically varying transcription and preprocessing methods while holding the retrieval mechanism fixed, we demonstrate that BM25 can recover large gaps on multilingual and visual benchmarks. Our findings call for decomposed evaluation benchmarks that separately measure transcription and retrieval capabilities, enabling the field to correctly attribute progress and focus effort where it matters.