Zero-shot data citation function classification using transformer-based large language models (LLMs)
This addresses the need for scalable data citation analysis in genomics, though it is incremental as it builds on existing LLM capabilities.
The authors tackled the problem of automatically classifying how or why datasets are used in scientific publications by applying the Llama 3.1-405B LLM in a zero-shot setting, achieving an F1 score of 0.674 without predefined categories.
Efforts have increased in recent years to identify associations between specific datasets and the scientific literature that incorporates them. Knowing that a given publication cites a given dataset, the next logical step is to explore how or why that data was used. Advances in recent years with pretrained, transformer-based large language models (LLMs) offer potential means for scaling the description of data use cases in the published literature. This avoids expensive manual labeling and the development of training datasets for classical machine-learning (ML) systems. In this work we apply an open-source LLM, Llama 3.1-405B, to generate structured data use case labels for publications known to incorporate specific genomic datasets. We also introduce a novel evaluation framework for determining the efficacy of our methods. Our results demonstrate that the stock model can achieve an F1 score of .674 on a zero-shot data citation classification task with no previously defined categories. While promising, our results are qualified by barriers related to data availability, prompt overfitting, computational infrastructure, and the expense required to conduct responsible performance evaluation.