Robust Layout-aware IE for Visually Rich Documents with Pre-trained Language Models
This addresses the problem of extracting structured information from business documents for NLP and IR applications, with incremental improvements in performance and data efficiency.
The paper tackles information extraction from visually rich documents like invoices and resumes by combining pre-trained language models with graph neural networks to encode both text and layout, resulting in absolute F1 improvements of 6.3% on invoices and 4.7% on resumes over text-based baselines.
Many business documents processed in modern NLP and IR pipelines are visually rich: in addition to text, their semantics can also be captured by visual traits such as layout, format, and fonts. We study the problem of information extraction from visually rich documents (VRDs) and present a model that combines the power of large pre-trained language models and graph neural networks to efficiently encode both textual and visual information in business documents. We further introduce new fine-tuning objectives to improve in-domain unsupervised fine-tuning to better utilize large amount of unlabeled in-domain data. We experiment on real world invoice and resume data sets and show that the proposed method outperforms strong text-based RoBERTa baselines by 6.3% absolute F1 on invoices and 4.7% absolute F1 on resumes. When evaluated in a few-shot setting, our method requires up to 30x less annotation data than the baseline to achieve the same level of performance at ~90% F1.