Non-parametric, Nearest-neighbor-assisted Fine-tuning for Neural Machine Translation
This work addresses the challenge of enhancing translation fluency, particularly for grammatical relations and function words, for users of neural machine translation systems, representing an incremental advancement in fine-tuning methods.
The paper tackled the problem of improving machine translation models during fine-tuning by incorporating non-parametric k-nearest-neighbor statistics to inform gradient updates, resulting in consistent improvements of up to 1.45 BLEU and 1.28 BLEU for German-English and English-German translations on standard datasets.
Non-parametric, k-nearest-neighbor algorithms have recently made inroads to assist generative models such as language models and machine translation decoders. We explore whether such non-parametric models can improve machine translation models at the fine-tuning stage by incorporating statistics from the kNN predictions to inform the gradient updates for a baseline translation model. There are multiple methods which could be used to incorporate kNN statistics and we investigate gradient scaling by a gating mechanism, the kNN's ground truth probability, and reinforcement learning. For four standard in-domain machine translation datasets, compared with classic fine-tuning, we report consistent improvements of all of the three methods by as much as 1.45 BLEU and 1.28 BLEU for German-English and English-German translations respectively. Through qualitative analysis, we found particular improvements when it comes to translating grammatical relations or function words, which results in increased fluency of our model.