A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends
It addresses the problem of automatically processing complex documents for researchers and practitioners, but is incremental as it synthesizes existing work rather than presenting new findings.
This survey reviews recent advancements in Multimodal Large Language Models (MLLMs) for Visually-Rich Document Understanding (VRDU), covering methods, training paradigms, and datasets, while identifying challenges and future directions to improve system efficiency, generalizability, and robustness.
Visually-Rich Document Understanding (VRDU) has emerged as a critical field, driven by the need to automatically process documents containing complex visual, textual, and layout information. Recently, Multimodal Large Language Models (MLLMs) have shown remarkable potential in this domain, leveraging both Optical Character Recognition (OCR)-dependent and OCR-free frameworks to extract and interpret information in document images. This survey reviews recent advancements in MLLM-based VRDU, highlighting three core components: (1) methods for encoding and fusing textual, visual, and layout features; (2) training paradigms, including pretraining strategies, instruction-response tuning, and the trainability of different model modules; and (3) datasets utilized for pretraining, instruction-tuning, and supervised fine-tuning. Finally, we discuss the challenges and opportunities in this evolving field and propose future directions to advance the efficiency, generalizability, and robustness of VRDU systems.