A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
This work addresses the problem of high computational cost in adversarial training for NLP practitioners, offering an efficient alternative with competitive performance.
The paper tackles the computational expense of adversarial training in NLP by proposing cutoff, a simple data augmentation method that erases parts of input sentences and uses a Jensen-Shannon Divergence loss for training. It achieves results on par or better than adversarial approaches on the GLUE benchmark and state-of-the-art BLEU scores on the IWSLT2014 German-English dataset.
Adversarial training has been shown effective at endowing the learned representations with stronger generalization ability. However, it typically requires expensive computation to determine the direction of the injected perturbations. In this paper, we introduce a set of simple yet effective data augmentation strategies dubbed cutoff, where part of the information within an input sentence is erased to yield its restricted views (during the fine-tuning stage). Notably, this process relies merely on stochastic sampling and thus adds little computational overhead. A Jensen-Shannon Divergence consistency loss is further utilized to incorporate these augmented samples into the training objective in a principled manner. To verify the effectiveness of the proposed strategies, we apply cutoff to both natural language understanding and generation problems. On the GLUE benchmark, it is demonstrated that cutoff, in spite of its simplicity, performs on par or better than several competitive adversarial-based approaches. We further extend cutoff to machine translation and observe significant gains in BLEU scores (based upon the Transformer Base model). Moreover, cutoff consistently outperforms adversarial training and achieves state-of-the-art results on the IWSLT2014 German-English dataset.