SoMeSci- A 5 Star Open Data Gold Standard Knowledge Graph of Software Mentions in Scientific Articles
This addresses the need for reliable ground truth data to improve information extraction in scholarly texts, particularly for researchers and developers working on Named Entity Recognition and related tasks, though it is incremental as it builds on existing annotation efforts.
The authors tackled the problem of automatically extracting and disambiguating software mentions in scientific articles by creating SoMeSci, a gold standard knowledge graph containing 3756 high-quality annotations (IRR: κ=.82) from 1367 PubMed Central articles, which includes relation labels and distinguishes between software types and mention types.
Knowledge about software used in scientific investigations is important for several reasons, for instance, to enable an understanding of provenance and methods involved in data handling. However, software is usually not formally cited, but rather mentioned informally within the scholarly description of the investigation, raising the need for automatic information extraction and disambiguation. Given the lack of reliable ground truth data, we present SoMeSci (Software Mentions in Science) a gold standard knowledge graph of software mentions in scientific articles. It contains high quality annotations (IRR: $κ{=}.82$) of 3756 software mentions in 1367 PubMed Central articles. Besides the plain mention of the software, we also provide relation labels for additional information, such as the version, the developer, a URL or citations. Moreover, we distinguish between different types, such as application, plugin or programming environment, as well as different types of mentions, such as usage or creation. To the best of our knowledge, SoMeSci is the most comprehensive corpus about software mentions in scientific articles, providing training samples for Named Entity Recognition, Relation Extraction, Entity Disambiguation, and Entity Linking. Finally, we sketch potential use cases and provide baseline results.