Regularizing Recurrent Neural Networks via Sequence Mixup
This work provides an incremental regularization method for improving the performance of RNNs in various sequence tasks.
This paper extends Input Mixup and Manifold Mixup regularization techniques to Recurrent Neural Networks (RNNs). Applying sequence mixup to a BiLSTM-CRF model for Named Entity Recognition on CoNLL-2003 data improved the F-1 score and reduced loss.
In this paper, we extend a class of celebrated regularization techniques originally proposed for feed-forward neural networks, namely Input Mixup (Zhang et al., 2017) and Manifold Mixup (Verma et al., 2018), to the realm of Recurrent Neural Networks (RNN). Our proposed methods are easy to implement and have a low computational complexity, while leverage the performance of simple neural architectures in a variety of tasks. We have validated our claims through several experiments on real-world datasets, and also provide an asymptotic theoretical analysis to further investigate the properties and potential impacts of our proposed techniques. Applying sequence mixup to BiLSTM-CRF model (Huang et al., 2015) to Named Entity Recognition task on CoNLL-2003 data (Sang and De Meulder, 2003) has improved the F-1 score on the test stage and reduced the loss, considerably.