Gradient-guided Loss Masking for Neural Machine Translation
This addresses data quality issues in machine translation, though it is incremental as it builds on existing gradient-based filtering approaches.
The paper tackles the problem of low-quality training data harming neural machine translation performance by dynamically masking harmful data during training based on gradient alignment with clean data, achieving significant improvements across three WMT language pairs with generalizable domain results.
To mitigate the negative effect of low quality training data on the performance of neural machine translation models, most existing strategies focus on filtering out harmful data before training starts. In this paper, we explore strategies that dynamically optimize data usage during the training process using the model's gradients on a small set of clean data. At each training step, our algorithm calculates the gradient alignment between the training data and the clean data to mask out data with negative alignment. Our method has a natural intuition: good training data should update the model parameters in a similar direction as the clean data. Experiments on three WMT language pairs show that our method brings significant improvement over strong baselines, and the improvements are generalizable across test data from different domains.