Benchmarking PDF Parsers on Table Extraction with LLM-based Semantic Evaluation
This work addresses the need for reliable table extraction in scientific data mining by providing a scalable evaluation method, though it is incremental as it builds on existing parser evaluation with a new metric.
The paper tackled the problem of evaluating table extraction from PDFs by introducing a benchmarking framework that uses LLM-based semantic evaluation, achieving a correlation with human judgment of Pearson r=0.93 compared to existing metrics like TEDS (r=0.68) and GriTS (r=0.70).
Reliably extracting tables from PDFs is essential for large-scale scientific data mining and knowledge base construction, yet existing evaluation approaches rely on rule-based metrics that fail to capture semantic equivalence of table content. We present a benchmarking framework based on synthetically generated PDFs with precise LaTeX ground truth, using tables sourced from arXiv to ensure realistic complexity and diversity. As our central methodological contribution, we apply LLM-as-a-judge for semantic table evaluation, integrated into a matching pipeline that accommodates inconsistencies in parser outputs. Through a human validation study comprising over 1,500 quality judgments on extracted table pairs, we show that LLM-based evaluation achieves substantially higher correlation with human judgment (Pearson r=0.93) compared to Tree Edit Distance-based Similarity (TEDS, r=0.68) and Grid Table Similarity (GriTS, r=0.70). Evaluating 21 contemporary PDF parsers across 100 synthetic documents containing 451 tables reveals significant performance disparities. Our results offer practical guidance for selecting parsers for tabular data extraction and establish a reproducible, scalable evaluation methodology for this critical task. Code and data: https://github.com/phorn1/pdf-parse-bench Metric study and human evaluation: https://github.com/phorn1/table-metric-study