Controllable Dual Skew Divergence Loss for Neural Machine Translation
This work addresses a specific issue in neural machine translation training, offering an incremental improvement for researchers and practitioners in the field.
The paper tackles the problem of neural machine translation models overgeneralizing and getting stuck in local optima when trained with cross-entropy loss, proposing a controllable dual skew divergence loss that improves performance on WMT 2014 English-German and English-French tasks.
In sequence prediction tasks like neural machine translation, training with cross-entropy loss often leads to models that overgeneralize and plunge into local optima. In this paper, we propose an extended loss function called \emph{dual skew divergence} (DSD) that integrates two symmetric terms on KL divergences with a balanced weight. We empirically discovered that such a balanced weight plays a crucial role in applying the proposed DSD loss into deep models. Thus we eventually develop a controllable DSD loss for general-purpose scenarios. Our experiments indicate that switching to the DSD loss after the convergence of ML training helps models escape local optima and stimulates stable performance improvements. Our evaluations on the WMT 2014 English-German and English-French translation tasks demonstrate that the proposed loss as a general and convenient mean for NMT training indeed brings performance improvement in comparison to strong baselines.