Text Smoothing: Enhance Various Data Augmentation Methods on Text Classification Tasks
This addresses data efficiency for text classification tasks, though it is incremental as it builds on existing augmentation techniques.
The paper tackles the problem of data scarcity in text classification by proposing text smoothing, which replaces one-hot token representations with smoothed distributions from a masked language model, resulting in substantial performance gains over existing augmentation methods in low-resource settings.
Before entering the neural network, a token is generally converted to the corresponding one-hot representation, which is a discrete distribution of the vocabulary. Smoothed representation is the probability of candidate tokens obtained from a pre-trained masked language model, which can be seen as a more informative substitution to the one-hot representation. We propose an efficient data augmentation method, termed text smoothing, by converting a sentence from its one-hot representation to a controllable smoothed representation. We evaluate text smoothing on different benchmarks in a low-resource regime. Experimental results show that text smoothing outperforms various mainstream data augmentation methods by a substantial margin. Moreover, text smoothing can be combined with those data augmentation methods to achieve better performance.