Large Language Models in the Task of Automatic Validation of Text Classifier Predictions
This addresses the problem of high annotation costs and limited specialist availability for machine learning practitioners, though it is incremental as it applies existing LLMs to a known bottleneck.
The paper tackles the labor-intensive and costly process of human annotation for validating text classifier predictions, especially in incremental learning scenarios, by proposing to use Large Language Models (LLMs) to automate this validation, thereby supporting model quality and efficient retraining.
Machine learning models for text classification are trained to predict a class for a given text. To do this, training and validation samples must be prepared: a set of texts is collected, and each text is assigned a class. These classes are usually assigned by human annotators with different expertise levels, depending on the specific classification task. Collecting such samples from scratch is labor-intensive because it requires finding specialists and compensating them for their work; moreover, the number of available specialists is limited, and their productivity is constrained by human factors. While it may not be too resource-intensive to collect samples once, the ongoing need to retrain models (especially in incremental learning pipelines) to address data drift (also called model drift) makes the data collection process crucial and costly over the model's entire lifecycle. This paper proposes several approaches to replace human annotators with Large Language Models (LLMs) to test classifier predictions for correctness, helping ensure model quality and support high-quality incremental learning.