DocumentCLIP: Linking Figures and Main Body Text in Reflowed Documents
This addresses multimodal document understanding for applications like news articles and product descriptions, but it is incremental as it extends contrastive learning to intra-document links.
The paper tackles the problem of aligning multiple images with longer text within documents, which existing vision-language models ignore, and shows that DocumentCLIP outperforms state-of-the-art baselines in supervised settings and achieves the best zero-shot performance after human evaluation.
Vision-language pretraining models have achieved great success in supporting multimedia applications by understanding the alignments between images and text. While existing vision-language pretraining models primarily focus on understanding single image associated with a single piece of text, they often ignore the alignment at the intra-document level, consisting of multiple sentences with multiple images. In this work, we propose DocumentCLIP, a salience-aware contrastive learning framework to enforce vision-language pretraining models to comprehend the interaction between images and longer text within documents. Our model is beneficial for the real-world multimodal document understanding like news article, magazines, product descriptions, which contain linguistically and visually richer content. To the best of our knowledge, we are the first to explore multimodal intra-document links by contrastive learning. In addition, we collect a large Wikipedia dataset for pretraining, which provides various topics and structures. Experiments show DocumentCLIP not only outperforms the state-of-the-art baselines in the supervised setting, but also achieves the best zero-shot performance in the wild after human evaluation. Our code is available at https://github.com/FuxiaoLiu/DocumentCLIP.