AI for Monitoring and Classifying Data Used in Research Literature
This work is significant for researchers and institutions needing to track the impact and usage of datasets, contributing to transparency and reproducibility in research.
The paper addresses the lack of infrastructure for monitoring dataset usage in research literature by introducing a multitask GLiNER-based framework. This framework jointly extracts dataset mentions, identifies relations, and classifies usage context, leveraging synthetic data generation and LLM-based revalidation to overcome label scarcity and improve reliability and consistency.
While platforms like Google Scholar and Semantic Scholar track citations for academic papers, no comparable infrastructure exists for monitoring dataset usage in research literature, leaving the landscape of data use largely opaque. Addressing this gap is critical for transparency, reproducibility, and monitoring of impact, yet progress is hindered by inconsistent citation practices, scarce labeled data, and ambiguous references to datasets in the wild. Traditional NLP approaches struggle with these challenges, motivating the shift toward more adaptive, semantically rich models. Building on prior work using LLMs for data mention detection and synthetic data for bootstrapping training, this paper presents an updated methodology for scalable dataset monitoring. We introduce a multitask GLiNER-based framework that jointly performs dataset mention extraction, relation identification, and usage-context classification. To address label scarcity, the pipeline leverages synthetic data generation to produce training examples and LLM-based revalidation to filter incorrect mentions and enforce labeling consistency, together improving reliability, coverage, and output consistency across the training pipeline. This work advances the development of open-source tools for monitoring data use in research literature, contributing to the broader goal of generalizable, unconstrained dataset citation tracking.