DTBench: A Synthetic Benchmark for Document-to-Table Extraction
This benchmark provides a comprehensive testbed for researchers and developers working on Doc2Table extraction, enabling systematic evaluation and facilitating future research by highlighting specific areas of weakness in current LLMs.
This paper introduces DTBench, a synthetic benchmark for Document-to-Table (Doc2Table) extraction, which addresses the limitations of existing benchmarks in evaluating LLMs' ability to produce precisely structured tables, especially for indirect extraction requiring complex reasoning. DTBench, generated using a reverse Table2Doc paradigm and multi-agent synthesis, covers 5 major categories and 13 subcategories of Doc2Table capabilities. Evaluations on mainstream LLMs using DTBench reveal significant performance gaps and persistent challenges in reasoning, faithfulness, and conflict resolution.
Document-to-table (Doc2Table) extraction derives structured tables from unstructured documents under a target schema, enabling reliable and verifiable SQL-based data analytics. Although large language models (LLMs) have shown promise in flexible information extraction, their ability to produce precisely structured tables remains insufficiently understood, particularly for indirect extraction that requires complex capabilities such as reasoning and conflict resolution. Existing benchmarks neither explicitly distinguish nor comprehensively cover the diverse capabilities required in Doc2Table extraction. We argue that a capability-aware benchmark is essential for systematic evaluation. However, constructing such benchmarks using human-annotated document-table pairs is costly, difficult to scale, and limited in capability coverage. To address this, we adopt a reverse Table2Doc paradigm and design a multi-agent synthesis workflow to generate documents from ground-truth tables. Based on this approach, we present DTBench, a synthetic benchmark that adopts a proposed two-level taxonomy of Doc2Table capabilities, covering 5 major categories and 13 subcategories. We evaluate several mainstream LLMs on DTBench, and demonstrate substantial performance gaps across models, as well as persistent challenges in reasoning, faithfulness, and conflict resolution. DTBench provides a comprehensive testbed for data generation and evaluation, facilitating future research on Doc2Table extraction. The benchmark is publicly available at https://github.com/ZJU-DAILY/DTBench.