Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction
This addresses the challenge of structured knowledge extraction from fragmented documents for users relying on LLMs for data organization, though it is incremental as it focuses on benchmarking rather than a new method.
The authors tackled the problem of large language models (LLMs) generating disorganized outputs when extracting information from complex documents, by introducing the AOE benchmark to evaluate their ability to reconstruct isolated information into organized tables. The results showed that even state-of-the-art LLMs struggled significantly on this benchmark.
With the emergence of large language models (LLMs), there is an expectation that LLMs can effectively extract explicit information from complex real-world documents (e.g., papers, reports). However, most LLMs generate paragraph-style answers that are chaotic, disorganized, and untraceable. To bridge this gap, we introduce the Arranged and Organized Extraction Benchmark (AOE), a new bilingual benchmark with data and documents of varying lengths designed to systematically evaluate the ability of LLMs to comprehend fragmented documents and reconstruct isolated information into one organized table. Unlike conventional text-to-table tasks, which rely on fixed schema and narrow task domains, AOE includes 11 carefully crafted tasks across three diverse domains, requiring models to generate context-specific schema tailored to varied input queries. In the experiment, we evaluated both open-source and closed-source state-of-the-art LLMs. The results show that even the most advanced models struggled significantly. The benchmark is available at https://anonymous.4open.science/r/AOE-Benchmark/.