Document Intelligence Metrics for Visually Rich Document Evaluation
This work addresses the need for standardized evaluation metrics in Document Intelligence for researchers and practitioners, but it is incremental as it focuses on creating a library rather than advancing model performance.
The authors tackled the problem of evaluating information extraction models for Visually-Rich Documents (VRDs) by introducing DI-Metrics, a Python library that includes text-based, geometric-based, and hierarchical metrics, and they applied it to the CORD dataset to compare three SOTA models and one industry model, though no specific performance numbers were provided.
The processing of Visually-Rich Documents (VRDs) is highly important in information extraction tasks associated with Document Intelligence. We introduce DI-Metrics, a Python library devoted to VRD model evaluation comprising text-based, geometric-based and hierarchical metrics for information extraction tasks. We apply DI-Metrics to evaluate information extraction performance using publicly available CORD dataset, comparing performance of three SOTA models and one industry model. The open-source library is available on GitHub.