Collage: Decomposable Rapid Prototyping for Information Extraction on Scientific PDFs
This tool addresses the challenge for scientists outside NLP to evaluate and apply information extraction systems to their own domains, though it is incremental as it builds on existing models and formats.
The paper tackles the difficulty of comparing and applying diverse information extraction models to scientific PDFs by introducing Collage, a tool for rapid prototyping, visualization, and evaluation that supports various models and provides debugging capabilities, demonstrated in materials science literature review.
Recent years in NLP have seen the continued development of domain-specific information extraction tools for scientific documents, alongside the release of increasingly multimodal pretrained transformer models. While the opportunity for scientists outside of NLP to evaluate and apply such systems to their own domains has never been clearer, these models are difficult to compare: they accept different input formats, are often black-box and give little insight into processing failures, and rarely handle PDF documents, the most common format of scientific publication. In this work, we present Collage, a tool designed for rapid prototyping, visualization, and evaluation of different information extraction models on scientific PDFs. Collage allows the use and evaluation of any HuggingFace token classifier, several LLMs, and multiple other task-specific models out of the box, and provides extensible software interfaces to accelerate experimentation with new models. Further, we enable both developers and users of NLP-based tools to inspect, debug, and better understand modeling pipelines by providing granular views of intermediate states of processing. We demonstrate our system in the context of information extraction to assist with literature review in materials science.