LLMs vs Established Text Augmentation Techniques for Classification: When do the Benefits Outweight the Costs?
This work addresses the cost-benefit trade-off of using LLMs for data augmentation in classification tasks, providing practical guidance for researchers and practitioners, though it is incremental in evaluating existing methods.
The study compared LLM-based text augmentation with established methods across 6 datasets, 3 classifiers, and 2 fine-tuning methods, finding that LLM methods are only cost-beneficial with very few seeds and often yield similar or worse model accuracies.
The generative large language models (LLMs) are increasingly being used for data augmentation tasks, where text samples are LLM-paraphrased and then used for classifier fine-tuning. However, a research that would confirm a clear cost-benefit advantage of LLMs over more established augmentation methods is largely missing. To study if (and when) is the LLM-based augmentation advantageous, we compared the effects of recent LLM augmentation methods with established ones on 6 datasets, 3 classifiers and 2 fine-tuning methods. We also varied the number of seeds and collected samples to better explore the downstream model accuracy space. Finally, we performed a cost-benefit analysis and show that LLM-based methods are worthy of deployment only when very small number of seeds is used. Moreover, in many cases, established methods lead to similar or better model accuracies.