CiteWorth: Cite-Worthiness Detection for Improved Scientific Document Understanding
This work addresses the problem of limited annotated data for scientific text by leveraging citations to build a dataset, though it is incremental as it builds on existing methods like Longformer.
The authors tackled the challenge of scientific document understanding by creating CiteWorth, a large, cleaned dataset for cite-worthiness detection, and showed that their best model improved F1 by 5 points over SciBERT and that fine-tuning with this task enhances downstream performance.
Scientific document understanding is challenging as the data is highly domain specific and diverse. However, datasets for tasks with scientific text require expensive manual annotation and tend to be small and limited to only one or a few fields. At the same time, scientific documents contain many potential training signals, such as citations, which can be used to build large labelled datasets. Given this, we present an in-depth study of cite-worthiness detection in English, where a sentence is labelled for whether or not it cites an external source. To accomplish this, we introduce CiteWorth, a large, contextualized, rigorously cleaned labelled dataset for cite-worthiness detection built from a massive corpus of extracted plain-text scientific documents. We show that CiteWorth is high-quality, challenging, and suitable for studying problems such as domain adaptation. Our best performing cite-worthiness detection model is a paragraph-level contextualized sentence labelling model based on Longformer, exhibiting a 5 F1 point improvement over SciBERT which considers only individual sentences. Finally, we demonstrate that language model fine-tuning with cite-worthiness as a secondary task leads to improved performance on downstream scientific document understanding tasks.