CCpdf: Building a High Quality Corpus for Visually Rich Documents from Web Crawl Data
This provides researchers with a high-quality, accessible resource to develop better multilingual language models for document understanding, though it is incremental as it builds on existing web data and processing methods.
The paper tackles the lack of diverse, multilingual, and publicly available corpora for document understanding by proposing an efficient pipeline to build a large-scale corpus of PDF files from web crawl data, resulting in the CCpdf dataset and tools for language model pretraining.
In recent years, the field of document understanding has progressed a lot. A significant part of this progress has been possible thanks to the use of language models pretrained on large amounts of documents. However, pretraining corpora used in the domain of document understanding are single domain, monolingual, or nonpublic. Our goal in this paper is to propose an efficient pipeline for creating a big-scale, diverse, multilingual corpus of PDF files from all over the Internet using Common Crawl, as PDF files are the most canonical types of documents as considered in document understanding. We analysed extensively all of the steps of the pipeline and proposed a solution which is a trade-off between data quality and processing time. We also share a CCpdf corpus in a form or an index of PDF files along with a script for downloading them, which produces a collection useful for language model pretraining. The dataset and tools published with this paper offer researchers the opportunity to develop even better multilingual language models.