Scrambled Translation Problem: A Problem of Denoising UNMT
This addresses a specific error type in UNMT systems, improving translation quality for languages like English-French, English-German, English-Spanish, and Hindi-Punjabi, though it is incremental as it modifies an existing training approach.
The paper identifies the 'Scrambled Translation problem' in Unsupervised Neural Machine Translation (UNMT) systems, where word shuffle noise causes correct words to be generated but not properly stitched into phrases, reducing BLEU scores. It proposes a retraining strategy that stops and resumes training without noise, achieving significant BLEU improvements on four language pairs.
In this paper, we identify an interesting kind of error in the output of Unsupervised Neural Machine Translation (UNMT) systems like \textit{Undreamt}(footnote). We refer to this error type as \textit{Scrambled Translation problem}. We observe that UNMT models which use \textit{word shuffle} noise (as in case of Undreamt) can generate correct words, but fail to stitch them together to form phrases. As a result, words of the translated sentence look \textit{scrambled}, resulting in decreased BLEU. We hypothesise that the reason behind \textit{scrambled translation problem} is 'shuffling noise' which is introduced in every input sentence as a denoising strategy. To test our hypothesis, we experiment by retraining UNMT models with a simple \textit{retraining} strategy. We stop the training of the Denoising UNMT model after a pre-decided number of iterations and resume the training for the remaining iterations -- which number is also pre-decided -- using original sentence as input without adding any noise. Our proposed solution achieves significant performance improvement UNMT models that train conventionally. We demonstrate these performance gains on four language pairs, \textit{viz.}, English-French, English-German, English-Spanish, Hindi-Punjabi. Our qualitative and quantitative analysis shows that the retraining strategy helps achieve better alignment as observed by attention heatmap and better phrasal translation, leading to statistically significant improvement in BLEU scores.