Comparing LLM Text Annotation Skills: A Study on Human Rights Violations in Social Media Data
It addresses the reliability of LLMs for sensitive, domain-specific tasks in multilingual contexts, but is incremental as it applies existing methods to new data.
This study compared multiple state-of-the-art LLMs (GPT-3.5, GPT-4, LLAMA3, Mistral 7B, and Claude-2) for zero-shot and few-shot annotation of human rights violations in Russian and Ukrainian social media posts, finding insights into their performance and error patterns against a human-annotated gold standard of 1000 samples.
In the era of increasingly sophisticated natural language processing (NLP) systems, large language models (LLMs) have demonstrated remarkable potential for diverse applications, including tasks requiring nuanced textual understanding and contextual reasoning. This study investigates the capabilities of multiple state-of-the-art LLMs - GPT-3.5, GPT-4, LLAMA3, Mistral 7B, and Claude-2 - for zero-shot and few-shot annotation of a complex textual dataset comprising social media posts in Russian and Ukrainian. Specifically, the focus is on the binary classification task of identifying references to human rights violations within the dataset. To evaluate the effectiveness of these models, their annotations are compared against a gold standard set of human double-annotated labels across 1000 samples. The analysis includes assessing annotation performance under different prompting conditions, with prompts provided in both English and Russian. Additionally, the study explores the unique patterns of errors and disagreements exhibited by each model, offering insights into their strengths, limitations, and cross-linguistic adaptability. By juxtaposing LLM outputs with human annotations, this research contributes to understanding the reliability and applicability of LLMs for sensitive, domain-specific tasks in multilingual contexts. It also sheds light on how language models handle inherently subjective and context-dependent judgments, a critical consideration for their deployment in real-world scenarios.