Scaling Public Health Text Annotation: Zero-Shot Learning vs. Crowdsourcing for Improved Efficiency and Labeling Accuracy
This work addresses the challenge of efficient and accurate text annotation for public health researchers using social media data, though it is incremental in comparing existing methods.
This study tackled the problem of labor-intensive manual labeling of social media data for public health research by comparing zero-shot labeling using large language models (LLMs) to crowd-sourced annotation for Twitter posts on sleep disorders, physical activity, and sedentary behavior. The results showed that LLMs can rival human performance in straightforward classification tasks and significantly reduce labeling time, but their accuracy decreases for tasks requiring nuanced domain knowledge.
Public health researchers are increasingly interested in using social media data to study health-related behaviors, but manually labeling this data can be labor-intensive and costly. This study explores whether zero-shot labeling using large language models (LLMs) can match or surpass conventional crowd-sourced annotation for Twitter posts related to sleep disorders, physical activity, and sedentary behavior. Multiple annotation pipelines were designed to compare labels produced by domain experts, crowd workers, and LLM-driven approaches under varied prompt-engineering strategies. Our findings indicate that LLMs can rival human performance in straightforward classification tasks and significantly reduce labeling time, yet their accuracy diminishes for tasks requiring more nuanced domain knowledge. These results clarify the trade-offs between automated scalability and human expertise, demonstrating conditions under which LLM-based labeling can be efficiently integrated into public health research without undermining label quality.