Repurposing Annotation Guidelines to Instruct LLM Annotators: A Case Study
This work addresses the need for scalable and cost-effective automated annotation in NLP, though it appears incremental as it builds on existing guidelines and LLM capabilities.
This study tackled the problem of adapting existing annotation guidelines for human annotators to instruct large language model (LLM) annotators in text annotation tasks, proposing a moderation-oriented repurposing method and demonstrating its effectiveness in a case study with the NCBI Disease Corpus, while identifying practical challenges.
This study investigates how existing annotation guidelines can be repurposed to instruct large language model (LLM) annotators for text annotation tasks. Traditional guidelines are written for human annotators who internalize training, while LLMs require explicit, structured instructions. We propose a moderation-oriented guideline repurposing method that transforms guidelines into clear directives for LLMs through an LLM moderation process. Using the NCBI Disease Corpus as a case study, our experiments show that repurposed guidelines can effectively guide LLM annotators, while revealing several practical challenges. The results highlight the potential of this workflow to support scalable and cost-effective refinement of annotation guidelines and automated annotation.