GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation
This addresses data scarcity issues in text classification tasks, though it appears incremental as it builds on existing language model capabilities.
The paper tackles the problem of data and inference scalability in prompt-based classification by proposing a novel data augmentation technique that uses large-scale language models to generate realistic text samples and soft-labels, showing huge performance improvements over existing methods.
Large-scale language models such as GPT-3 are excellent few-shot learners, allowing them to be controlled via natural text prompts. Recent studies report that prompt-based direct classification eliminates the need for fine-tuning but lacks data and inference scalability. This paper proposes a novel data augmentation technique that leverages large-scale language models to generate realistic text samples from a mixture of real samples. We also propose utilizing soft-labels predicted by the language models, effectively distilling knowledge from the large-scale language models and creating textual perturbations simultaneously. We perform data augmentation experiments on diverse classification tasks and show that our method hugely outperforms existing text augmentation methods. Ablation studies and a qualitative analysis provide more insights into our approach.