LlamBERT: Large-scale low-cost data annotation in NLP
This addresses cost reduction for NLP practitioners, but it is incremental as it builds on existing LLM and transformer methods.
The paper tackled the problem of high costs in using Large Language Models (LLMs) for NLP tasks by proposing LlamBERT, a hybrid method that uses LLMs to annotate small subsets of unlabeled data for fine-tuning transformer encoders, resulting in slightly lower accuracy but significantly improved cost-effectiveness, as evaluated on datasets like IMDb and UMLS.
Large Language Models (LLMs), such as GPT-4 and Llama 2, show remarkable proficiency in a wide range of natural language processing (NLP) tasks. Despite their effectiveness, the high costs associated with their use pose a challenge. We present LlamBERT, a hybrid approach that leverages LLMs to annotate a small subset of large, unlabeled databases and uses the results for fine-tuning transformer encoders like BERT and RoBERTa. This strategy is evaluated on two diverse datasets: the IMDb review dataset and the UMLS Meta-Thesaurus. Our results indicate that the LlamBERT approach slightly compromises on accuracy while offering much greater cost-effectiveness.