Self-Guided Curriculum Learning for Neural Machine Translation
This work addresses translation quality for NMT users, but it is incremental as it builds on existing curriculum learning methods with a novel difficulty metric.
The paper tackles the problem of improving neural machine translation by proposing a self-guided curriculum learning strategy that uses the model's own BLEU score to measure learning difficulty, resulting in consistent performance gains on benchmarks like WMT14 English-German and WMT17 Chinese-English.
In the field of machine learning, the well-trained model is assumed to be able to recover the training labels, i.e. the synthetic labels predicted by the model should be as close to the ground-truth labels as possible. Inspired by this, we propose a self-guided curriculum strategy to encourage the learning of neural machine translation (NMT) models to follow the above recovery criterion, where we cast the recovery degree of each training example as its learning difficulty. Specifically, we adopt the sentence level BLEU score as the proxy of recovery degree. Different from existing curricula relying on linguistic prior knowledge or third-party language models, our chosen learning difficulty is more suitable to measure the degree of knowledge mastery of the NMT models. Experiments on translation benchmarks, including WMT14 English$\Rightarrow$German and WMT17 Chinese$\Rightarrow$English, demonstrate that our approach can consistently improve translation performance against strong baseline Transformer.