Improving reference mining in patents with BERT
This work provides a more comprehensive dataset of science-industry links for researchers studying innovation and technology transfer, improving upon existing methods that primarily rely on front-page citations.
This paper addresses the problem of extracting scientific references from patents, which is crucial for understanding science-industry connections. By improving training data and utilizing BERT-based models, the authors achieved recall scores of around 97% and extracted 50% more references (735,000 in total) from 33,000 patents compared to previous methods.
In this paper we address the challenge of extracting scientific references from patents. We approach the problem as a sequence labelling task and investigate the merits of BERT models to the extraction of these long sequences. References in patents to scientific literature are relevant to study the connection between science and industry. Most prior work only uses the front-page citations for this analysis, which are provided in the metadata of patent archives. In this paper we build on prior work using Conditional Random Fields (CRF) and Flair for reference extraction. We improve the quality of the training data and train three BERT-based models on the labelled data (BERT, bioBERT, sciBERT). We find that the improved training data leads to a large improvement in the quality of the trained models. In addition, the BERT models beat CRF and Flair, with recall scores around 97% obtained with cross validation. With the best model we label a large collection of 33 thousand patents, extract the citations, and match them to publications in the Web of Science database. We extract 50% more references than with the old training data and methods: 735 thousand references in total. With these patent-publication links, follow-up research will further analyze which types of scientific work lead to inventions.