VRDU: A Benchmark for Visually-rich Document Understanding
This provides a more comprehensive benchmark for researchers and practitioners working on automating business workflows through document understanding, though it is incremental as it builds on existing multi-modal models.
The authors tackled the lack of benchmarks reflecting real-world complexity in visually-rich document understanding by introducing VRDU, a benchmark with two datasets that show strong baselines struggle with generalization to new templates, few-shot learning, and hierarchical fields.
Understanding visually-rich business documents to extract structured data and automate business workflows has been receiving attention both in academia and industry. Although recent multi-modal language models have achieved impressive results, we find that existing benchmarks do not reflect the complexity of real documents seen in industry. In this work, we identify the desiderata for a more comprehensive benchmark and propose one we call Visually Rich Document Understanding (VRDU). VRDU contains two datasets that represent several challenges: rich schema including diverse data types as well as hierarchical entities, complex templates including tables and multi-column layouts, and diversity of different layouts (templates) within a single document type. We design few-shot and conventional experiment settings along with a carefully designed matching algorithm to evaluate extraction results. We report the performance of strong baselines and offer three observations: (1) generalizing to new document templates is still very challenging, (2) few-shot performance has a lot of headroom, and (3) models struggle with hierarchical fields such as line-items in an invoice. We plan to open source the benchmark and the evaluation toolkit. We hope this helps the community make progress on these challenging tasks in extracting structured data from visually rich documents.