OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations
This addresses the need for better evaluation standards in document parsing, which is critical for LLMs and RAG systems, but it is incremental as it builds on existing benchmarking efforts.
The authors tackled the lack of fair and comprehensive evaluation in document parsing by introducing OmniDocBench, a benchmark with diverse document types and multi-level annotations, which revealed strengths and weaknesses of existing methods across different document sources.
Document content extraction is a critical task in computer vision, underpinning the data needs of large language models (LLMs) and retrieval-augmented generation (RAG) systems. Despite recent progress, current document parsing methods have not been fairly and comprehensively evaluated due to the narrow coverage of document types and the simplified, unrealistic evaluation procedures in existing benchmarks. To address these gaps, we introduce OmniDocBench, a novel benchmark featuring high-quality annotations across nine document sources, including academic papers, textbooks, and more challenging cases such as handwritten notes and densely typeset newspapers. OmniDocBench supports flexible, multi-level evaluations--ranging from an end-to-end assessment to the task-specific and attribute--based analysis using 19 layout categories and 15 attribute labels. We conduct a thorough evaluation of both pipeline-based methods and end-to-end vision-language models, revealing their strengths and weaknesses across different document types. OmniDocBench sets a new standard for the fair, diverse, and fine-grained evaluation in document parsing. Dataset and code are available at https://github.com/opendatalab/OmniDocBench.