HIP: Hierarchical Point Modeling and Pre-training for Visual Information Extraction
This work addresses the challenge of end-to-end VIE for applications like document analysis, offering a unified framework that improves over OCR-dependent and OCR-free methods, though it appears incremental as it builds on existing models like CenterNet and DeepSolo.
The paper tackles the problem of integrating hierarchical subtasks in visual information extraction (VIE) by proposing HIP, which models entities as hierarchical points to address gaps between text spotting, word grouping, and entity labeling, resulting in outperforming previous state-of-the-art methods on public benchmarks.
End-to-end visual information extraction (VIE) aims at integrating the hierarchical subtasks of VIE, including text spotting, word grouping, and entity labeling, into a unified framework. Dealing with the gaps among the three subtasks plays a pivotal role in designing an effective VIE model. OCR-dependent methods heavily rely on offline OCR engines and inevitably suffer from OCR errors, while OCR-free methods, particularly those employing a black-box model, might produce outputs that lack interpretability or contain hallucinated content. Inspired by CenterNet, DeepSolo, and ESP, we propose HIP, which models entities as HIerarchical Points to better conform to the hierarchical nature of the end-to-end VIE task. Specifically, such hierarchical points can be flexibly encoded and subsequently decoded into desired text transcripts, centers of various regions, and categories of entities. Furthermore, we devise corresponding hierarchical pre-training strategies, categorized as image reconstruction, layout learning, and language enhancement, to reinforce the cross-modality representation of the hierarchical encoders. Quantitative experiments on public benchmarks demonstrate that HIP outperforms previous state-of-the-art methods, while qualitative results show its excellent interpretability.