Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understanding
This work addresses the challenge of fine-grained visual document understanding for applications like information extraction, representing an incremental advance by focusing on textline granularity.
The paper tackles the problem of fine-grained visual document understanding by proposing Wukong-Reader, a multi-modal pre-training method that leverages document textlines for improved alignment and representation, resulting in superior performance on various VDU tasks and enhanced localization ability.
Unsupervised pre-training on millions of digital-born or scanned documents has shown promising advances in visual document understanding~(VDU). While various vision-language pre-training objectives are studied in existing solutions, the document textline, as an intrinsic granularity in VDU, has seldom been explored so far. A document textline usually contains words that are spatially and semantically correlated, which can be easily obtained from OCR engines. In this paper, we propose Wukong-Reader, trained with new pre-training objectives to leverage the structural knowledge nested in document textlines. We introduce textline-region contrastive learning to achieve fine-grained alignment between the visual regions and texts of document textlines. Furthermore, masked region modeling and textline-grid matching are also designed to enhance the visual and layout representations of textlines. Experiments show that our Wukong-Reader has superior performance on various VDU tasks such as information extraction. The fine-grained alignment over textlines also empowers Wukong-Reader with promising localization ability.