MEGAnno+: A Human-LLM Collaborative Annotation System
This addresses the need for more accurate data labeling in NLP tasks, particularly for sociocultural or domain-specific contexts, but appears incremental as it builds on existing collaborative annotation ideas.
The paper tackles the problem of LLMs producing incorrect annotations for complex or domain-specific data by proposing MEGAnno+, a human-LLM collaborative annotation system, which aims to produce reliable and high-quality labels through effective management and verification.
Large language models (LLMs) can label data faster and cheaper than humans for various NLP tasks. Despite their prowess, LLMs may fall short in understanding of complex, sociocultural, or domain-specific context, potentially leading to incorrect annotations. Therefore, we advocate a collaborative approach where humans and LLMs work together to produce reliable and high-quality labels. We present MEGAnno+, a human-LLM collaborative annotation system that offers effective LLM agent and annotation management, convenient and robust LLM annotation, and exploratory verification of LLM labels by humans.