LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding
This work addresses the problem of improving text-layout interactions for researchers and practitioners in document understanding, representing an incremental advance with novel masking techniques.
The paper tackled the challenge of fusing text, layout, and image modalities in visually-rich document understanding by proposing LayoutMask, a multi-modal pre-training model that uses local 1D positions and novel masking strategies, achieving state-of-the-art results on tasks like form understanding and document image classification.
Visually-rich Document Understanding (VrDU) has attracted much research attention over the past years. Pre-trained models on a large number of document images with transformer-based backbones have led to significant performance gains in this field. The major challenge is how to fusion the different modalities (text, layout, and image) of the documents in a unified model with different pre-training tasks. This paper focuses on improving text-layout interactions and proposes a novel multi-modal pre-training model, LayoutMask. LayoutMask uses local 1D position, instead of global 1D position, as layout input and has two pre-training objectives: (1) Masked Language Modeling: predicting masked tokens with two novel masking strategies; (2) Masked Position Modeling: predicting masked 2D positions to improve layout representation learning. LayoutMask can enhance the interactions between text and layout modalities in a unified model and produce adaptive and robust multi-modal representations for downstream tasks. Experimental results show that our proposed method can achieve state-of-the-art results on a wide variety of VrDU problems, including form understanding, receipt understanding, and document image classification.