SCORE: A Semantic Evaluation Framework for Generative Document Parsing
This addresses the problem of fair and practical benchmarking for researchers and practitioners working with modern generative document parsing systems, offering a novel evaluation framework rather than an incremental improvement.
The paper tackles the problem that conventional evaluation metrics misclassify semantically correct but structurally divergent outputs from generative document parsing systems as errors, introducing SCORE as a new framework that integrates adjusted edit distance, token-level diagnostics, table evaluation with spatial tolerance, and hierarchy-aware consistency checks. Across 1,114 pages, SCORE revealed cross-dataset performance patterns missed by standard metrics, corrected distorted rankings in 2-5% of ambiguous cases where traditional metrics penalized systems by 12-25% on average, and reproduced traditional scores like table F1 up to 0.93 without requiring object-detection pipelines.
Multi-modal generative document parsing systems challenge traditional evaluation: unlike deterministic OCR or layout models, they often produce semantically correct yet structurally divergent outputs. Conventional metrics-CER, WER, IoU, or TEDS-misclassify such diversity as error, penalizing valid interpretations and obscuring system behavior. We introduce SCORE (Structural and COntent Robust Evaluation), an interpretation-agnostic framework that integrates (i) adjusted edit distance for robust content fidelity, (ii) token-level diagnostics to distinguish hallucinations from omissions, (iii) table evaluation with spatial tolerance and semantic alignment, and (iv) hierarchy-aware consistency checks. Together, these dimensions enable evaluation that embraces representational diversity while enforcing semantic rigor. Across 1,114 pages spanning a holistic benchmark and a field dataset, SCORE consistently revealed cross-dataset performance patterns missed by standard metrics. In 2-5% of pages with ambiguous table structures, traditional metrics penalized systems by 12-25% on average, leading to distorted rankings. SCORE corrected these cases, recovering equivalence between alternative but valid interpretations. Moreover, by normalizing generative outputs into a format-agnostic representation, SCORE reproduces traditional scores (e.g., table F1 up to 0.93) without requiring object-detection pipelines, demonstrating that generative parsing alone suffices for comprehensive evaluation. By exposing how interpretive diversity impacts evaluation outcomes and providing multi-dimensional, interpretable diagnostics, SCORE establishes foundational principles for semantically grounded, fair, and practical benchmarking of modern document parsing systems.