Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost
This work addresses generalization issues in NLP for domains with limited data, offering a cost-effective annotation method, though it is incremental as it builds on existing active learning and LLM annotation approaches.
The paper tackles the problem of NLP models failing on low-data domains by using large language models (LLMs) to annotate inputs, proposing a sampling strategy based on prediction score differences between base and finetuned models. Experiments show significant accuracy gains in classification and ranking tasks for both training and target domains.
State-of-the-art supervised NLP models achieve high accuracy but are also susceptible to failures on inputs from low-data regimes, such as domains that are not represented in training data. As an approximation to collecting ground-truth labels for the specific domain, we study the use of large language models (LLMs) for annotating inputs and improving the generalization of NLP models. Specifically, given a budget for LLM annotations, we present an algorithm for sampling the most informative inputs to annotate and retrain the NLP model. We find that popular active learning strategies such as uncertainty-based sampling do not work well. Instead, we propose a sampling strategy based on the difference in prediction scores between the base model and the finetuned NLP model, utilizing the fact that most NLP models are finetuned from a base model. Experiments with classification (semantic similarity) and ranking (semantic search) tasks show that our sampling strategy leads to significant gains in accuracy for both the training and target domains.