Corpus Conversion Service: A machine learning platform to ingest documents at scale [Poster abstract]
This addresses the need for scalable document ingestion systems to make content discoverable, though it appears incremental as it builds on existing machine learning techniques for document processing.
The paper tackles the problem of ingesting large volumes of scientific and technical documents, which are challenging due to formats like PDF and complex tables, by presenting a machine learning platform that allows training custom models; it achieves precision/recall results greater than 97% for conversion to structured formats and demonstrates scalability of its microservices.
Over the past few decades, the amount of scientific articles and technical literature has increased exponentially in size. Consequently, there is a great need for systems that can ingest these documents at scale and make their content discoverable. Unfortunately, both the format of these documents (e.g. the PDF format or bitmap images) as well as the presentation of the data (e.g. complex tables) make the extraction of qualitative and quantitive data extremely challenging. We present a platform to ingest documents at scale which is powered by Machine Learning techniques and allows the user to train custom models on document collections. We show precision/recall results greater than 97% with regard to conversion to structured formats, as well as scaling evidence for each of the microservices constituting the platform.