SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents
This dataset addresses the need for more comprehensive scientific information extraction, particularly for datasets, methods, and tasks, though it is incremental as it builds on existing SciIE datasets.
The authors tackled the problem of limited entity and relation extraction in scientific documents by releasing a new dataset with 106 manually annotated full-text publications, containing over 24k entities and 12k relations, which captures diverse mentions and interactions in context.
Scientific information extraction (SciIE) is critical for converting unstructured knowledge from scholarly articles into structured data (entities and relations). Several datasets have been proposed for training and validating SciIE models. However, due to the high complexity and cost of annotating scientific texts, those datasets restrict their annotations to specific parts of paper, such as abstracts, resulting in the loss of diverse entity mentions and relations in context. In this paper, we release a new entity and relation extraction dataset for entities related to datasets, methods, and tasks in scientific articles. Our dataset contains 106 manually annotated full-text scientific publications with over 24k entities and 12k relations. To capture the intricate use and interactions among entities in full texts, our dataset contains a fine-grained tag set for relations. Additionally, we provide an out-of-distribution test set to offer a more realistic evaluation. We conduct comprehensive experiments, including state-of-the-art supervised models and our proposed LLM-based baselines, and highlight the challenges presented by our dataset, encouraging the development of innovative models to further the field of SciIE.