UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis
This provides a benchmark for researchers and practitioners working on RAG applications in domains like academic literature and finance, though it is incremental as it focuses on evaluation rather than new methods.
They tackled the challenge of evaluating Retrieval-Augmented Generation (RAG) in real-world document analysis by introducing the UDA benchmark suite with 2,965 documents and 29,590 expert-annotated Q&A pairs, finding that data parsing and retrieval are critical for performance.
The use of Retrieval-Augmented Generation (RAG) has improved Large Language Models (LLMs) in collaborating with external data, yet significant challenges exist in real-world scenarios. In areas such as academic literature and finance question answering, data are often found in raw text and tables in HTML or PDF formats, which can be lengthy and highly unstructured. In this paper, we introduce a benchmark suite, namely Unstructured Document Analysis (UDA), that involves 2,965 real-world documents and 29,590 expert-annotated Q&A pairs. We revisit popular LLM- and RAG-based solutions for document analysis and evaluate the design choices and answer qualities across multiple document domains and diverse query types. Our evaluation yields interesting findings and highlights the importance of data parsing and retrieval. We hope our benchmark can shed light and better serve real-world document analysis applications. The benchmark suite and code can be found at https://github.com/qinchuanhui/UDA-Benchmark.