Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Survey
It provides a systematic overview for researchers and practitioners working on improving document retrieval in RAG systems, but it is incremental as it surveys existing methods without introducing new techniques.
This survey examines how Multimodal Large Language Models (MLLMs) address challenges in retrieving visually rich documents for retrieval-augmented generation (RAG), such as layout-dependent semantics and brittle OCR, by categorizing their roles into Modality-Unifying Captioners, Multimodal Embedders, and End-to-End Representers.
Visually rich documents (VRDs) challenge retrieval-augmented generation (RAG) with layout-dependent semantics, brittle OCR, and evidence spread across complex figures and structured tables. This survey examines how Multimodal Large Language Models (MLLMs) are being used to make VRD retrieval practical for RAG. We organize the literature into three roles: Modality-Unifying Captioners, Multimodal Embedders, and End-to-End Representers. We compare these roles along retrieval granularity, information fidelity, latency and index size, and compatibility with reranking and grounding. We also outline key trade-offs and offer some practical guidance on when to favor each role. Finally, we identify promising directions for future research, including adaptive retrieval units, model size reduction, and the development of evaluation methods.