OLAF: Towards Robust LLM-Based Annotation Framework in Empirical Software Engineering
This addresses the need for more robust annotation practices in software engineering research, though it is incremental as it builds on existing concerns without introducing new empirical results.
The paper tackles the problem of unreliable and non-reproducible LLM-based annotations in empirical software engineering by proposing OLAF, a conceptual framework that organizes key constructs like reliability and transparency to improve methodological standards.
Large Language Models (LLMs) are increasingly used in empirical software engineering (ESE) to automate or assist annotation tasks such as labeling commits, issues, and qualitative artifacts. Yet the reliability and reproducibility of such annotations remain underexplored. Existing studies often lack standardized measures for reliability, calibration, and drift, and frequently omit essential configuration details. We argue that LLM-based annotation should be treated as a measurement process rather than a purely automated activity. In this position paper, we outline the \textbf{Operationalization for LLM-based Annotation Framework (OLAF)}, a conceptual framework that organizes key constructs: \textit{reliability, calibration, drift, consensus, aggregation}, and \textit{transparency}. The paper aims to motivate methodological discussion and future empirical work toward more transparent and reproducible LLM-based annotation in software engineering research.