Magic Words or Methodical Work? Challenging Conventional Wisdom in LLM-Based Political Text Annotation
This work addresses methodological challenges for political scientists using LLMs for text annotation, highlighting the risk of researcher degrees of freedom and providing practical tools for more robust and reproducible research.
The study systematically evaluated how implementation choices like model selection, size, and prompt engineering affect LLM-based political text annotation, finding that interaction effects dominate and no single approach is best across tasks, with model size and popular techniques often unreliable. It developed a validation-first framework to guide researchers in making transparent decisions.
Political scientists are rapidly adopting large language models (LLMs) for text annotation, yet the sensitivity of annotation results to implementation choices remains poorly understood. Most evaluations test a single model or configuration; how model choice, model size, learning approach, and prompt style interact, and whether popular "best practices" survive controlled comparison, are largely unexplored. We present a controlled evaluation of these pipeline choices, testing six open-weight models across four political science annotation tasks under identical quantisation, hardware, and prompt-template conditions. Our central finding is methodological: interaction effects dominate main effects, so seemingly reasonable pipeline choices can become consequential researcher degrees of freedom. No single model, prompt style, or learning approach is uniformly superior, and the best-performing model varies across tasks. Two corollaries follow. First, model size is an unreliable guide both to cost and to performance: cross-family efficiency differences are so large that some larger models are less resource-intensive than much smaller alternatives, while within model families mid-range variants often match or exceed larger counterparts. Second, widely recommended prompt engineering techniques yield inconsistent and sometimes negative effects on annotation performance. We use these benchmark results to develop a validation-first framework - with a principled ordering of pipeline decisions, guidance on prompt freezing and held-out evaluation, reporting standards, and open-source tools - to help researchers navigate this decision space transparently.