Rethink the Effectiveness of Text Data Augmentation: An Empirical Analysis
This addresses the problem of optimizing language model performance for NLP practitioners, though it is incremental as it builds on existing data augmentation techniques.
The study tackled the debated impact of data augmentation on fine-tuning language models by evaluating three fine-tuning methods with back-translation across 7 NLP tasks, finding that continued pre-training on augmented data can improve performance by over 10% in few-shot settings.
In recent years, language models (LMs) have made remarkable progress in advancing the field of natural language processing (NLP). However, the impact of data augmentation (DA) techniques on the fine-tuning (FT) performance of these LMs has been a topic of ongoing debate. In this study, we evaluate the effectiveness of three different FT methods in conjugation with back-translation across an array of 7 diverse NLP tasks, including classification and regression types, covering single-sentence and sentence-pair tasks. Contrary to prior assumptions that DA does not contribute to the enhancement of LMs' FT performance, our findings reveal that continued pre-training on augmented data can effectively improve the FT performance of the downstream tasks. In the most favourable case, continued pre-training improves the performance of FT by more than 10% in the few-shot learning setting. Our finding highlights the potential of DA as a powerful tool for bolstering LMs' performance.