Enhancing Text Classification through LLM-Driven Active Learning and Human Annotation
This addresses the financial burden of annotation for text classification tasks, but it is incremental as it combines existing techniques like active learning and LLMs.
The study tackled the high cost of manual annotation in text classification by integrating human annotators and Large Language Models within an Active Learning framework, resulting in a substantial decrease in annotation costs while maintaining or improving model accuracy on three public datasets.
In the context of text classification, the financial burden of annotation exercises for creating training data is a critical issue. Active learning techniques, particularly those rooted in uncertainty sampling, offer a cost-effective solution by pinpointing the most instructive samples for manual annotation. Similarly, Large Language Models (LLMs) such as GPT-3.5 provide an alternative for automated annotation but come with concerns regarding their reliability. This study introduces a novel methodology that integrates human annotators and LLMs within an Active Learning framework. We conducted evaluations on three public datasets. IMDB for sentiment analysis, a Fake News dataset for authenticity discernment, and a Movie Genres dataset for multi-label classification.The proposed framework integrates human annotation with the output of LLMs, depending on the model uncertainty levels. This strategy achieves an optimal balance between cost efficiency and classification performance. The empirical results show a substantial decrease in the costs associated with data annotation while either maintaining or improving model accuracy.