A Human-Centered Workflow for Using Large Language Models in Content Analysis
This addresses the need for rigorous and transparent LLM applications in content analysis for researchers across disciplines like political science and sociology, though it is incremental as it builds on existing methodologies.
The paper tackles the problem of effectively using Large Language Models (LLMs) for content analysis by proposing a human-centered workflow that integrates LLMs into annotation, summarization, and information extraction tasks, with results including validation procedures and practical tools like a prompt library and Python code.
While many researchers use Large Language Models (LLMs) through chat-based access, their real potential lies in leveraging LLMs via application programming interfaces (APIs). This paper conceptualizes LLMs as universal text processing machines and presents a comprehensive workflow for employing LLMs in three qualitative and quantitative content analysis tasks: (1) annotation (an umbrella term for qualitative coding, labeling and text classification), (2) summarization, and (3) information extraction. The workflow is explicitly human-centered. Researchers design, supervise, and validate each stage of the LLM process to ensure rigor and transparency. Our approach synthesizes insights from extensive methodological literature across multiple disciplines: political science, sociology, computer science, psychology, and management. We outline validation procedures and best practices to address key limitations of LLMs, such as their black-box nature, prompt sensitivity, and tendency to hallucinate. To facilitate practical implementation, we provide supplementary materials, including a prompt library and Python code in Jupyter Notebook format, accompanied by detailed usage instructions.