A Large-Scale Dataset and Citation Intent Classification in Turkish with LLMs
This work provides a foundational dataset and robust framework for the Turkish NLP community, addressing a specific gap in qualitative citation analysis, though it is incremental as it builds on existing methods like DSPy and ensemble techniques.
The paper tackled the problem of classifying citation intents in Turkish, a challenging agglutinative language, by introducing a new dataset and a programmable classification pipeline using DSPy and stacked ensemble methods, achieving a state-of-the-art accuracy of 91.3%.
Understanding the qualitative intent of citations is essential for a comprehensive assessment of academic research, a task that poses unique challenges for agglutinative languages like Turkish. This paper introduces a systematic methodology and a foundational dataset to address this problem. We first present a new, publicly available dataset of Turkish citation intents, created with a purpose-built annotation tool. We then evaluate the performance of standard In-Context Learning (ICL) with Large Language Models (LLMs), demonstrating that its effectiveness is limited by inconsistent results caused by manually designed prompts. To address this core limitation, we introduce a programmable classification pipeline built on the DSPy framework, which automates prompt optimization systematically. For final classification, we employ a stacked generalization ensemble to aggregate outputs from multiple optimized models, ensuring stable and reliable predictions. This ensemble, with an XGBoost meta-model, achieves a state-of-the-art accuracy of 91.3\%. Ultimately, this study provides the Turkish NLP community and the broader academic circles with a foundational dataset and a robust classification framework paving the way for future qualitative citation studies.