Large Language Models as Automatic Annotators and Annotation Adjudicators for Fine-Grained Opinion Analysis
This work addresses the problem of dataset creation for fine-grained opinion analysis, reducing annotation costs and effort for researchers and practitioners in natural language processing, though it is incremental as it applies existing LLM capabilities to a specific annotation bottleneck.
The paper tackles the high cost and effort of human annotation for fine-grained opinion analysis by exploring large language models (LLMs) as automatic annotators and adjudicators, achieving high inter-annotator agreement across models for tasks like Aspect Sentiment Triplet Extraction and Aspect-Category-Opinion-Sentiment analysis.
Fine-grained opinion analysis of text provides a detailed understanding of expressed sentiments, including the addressed entity. Although this level of detail is sound, it requires considerable human effort and substantial cost to annotate opinions in datasets for training models, especially across diverse domains and real-world applications. We explore the feasibility of LLMs as automatic annotators for fine-grained opinion analysis, addressing the shortage of domain-specific labelled datasets. In this work, we use a declarative annotation pipeline. This approach reduces the variability of manual prompt engineering when using LLMs to identify fine-grained opinion spans in text. We also present a novel methodology for an LLM to adjudicate multiple labels and produce final annotations. After trialling the pipeline with models of different sizes for the Aspect Sentiment Triplet Extraction (ASTE) and Aspect-Category-Opinion-Sentiment (ACOS) analysis tasks, we show that LLMs can serve as automatic annotators and adjudicators, achieving high Inter-Annotator Agreement across individual LLM-based annotators. This reduces the cost and human effort needed to create these fine-grained opinion-annotated datasets.