Sequence-Level Mixed Sample Data Augmentation
This work provides an incremental improvement in data augmentation for sequence-to-sequence models, benefiting researchers and practitioners working on neural machine translation and compositional generalization.
This paper introduces SeqMix, a data augmentation technique that generates new training examples by softly combining existing input/output sequences. SeqMix consistently improves BLEU scores by approximately 1.0 on five translation datasets and shows further gains on tasks requiring compositional generalization like SCAN and semantic parsing.
Despite their empirical success, neural networks still have difficulty capturing compositional aspects of natural language. This work proposes a simple data augmentation approach to encourage compositional behavior in neural models for sequence-to-sequence problems. Our approach, SeqMix, creates new synthetic examples by softly combining input/output sequences from the training set. We connect this approach to existing techniques such as SwitchOut and word dropout, and show that these techniques are all approximating variants of a single objective. SeqMix consistently yields approximately 1.0 BLEU improvement on five different translation datasets over strong Transformer baselines. On tasks that require strong compositional generalization such as SCAN and semantic parsing, SeqMix also offers further improvements.