Dynamic Jointly Batch Selection for Data Efficient Machine Translation Fine-Tuning
This addresses data efficiency for machine translation practitioners, but it is incremental as it builds on existing fine-tuning and selection techniques.
The paper tackles the problem of data selection for fine-tuning machine translation models by introducing a method that uses a learnability score and batch selection to prioritize relevant examples, achieving up to a fivefold improvement in data efficiency and 24% computational efficiency gain compared to baselines.
Data quality and its effective selection are fundamental to improving the performance of machine translation models, serving as cornerstones for achieving robust and reliable translation systems. This paper presents a data selection methodology specifically designed for fine-tuning machine translation systems, which leverages the synergy between a learner model and a pre-trained reference model to enhance overall training effectiveness. By defining a learnability score, our approach systematically evaluates the utility of data points for training, ensuring that only the most relevant and impactful examples contribute to the fine-tuning process. Furthermore, our method employs a batch selection strategy which considers interdependencies among data points, optimizing the efficiency of the training process while maintaining a focus on data relevance. Experiments on English to Persian and several other language pairs using an mBART model fine-tuned on the CCMatrix dataset demonstrate that our method can achieve up to a fivefold improvement in data efficiency compared to an iid baseline. Experimental results indicate that our approach improves computational efficiency by 24 when utilizing cached embeddings, as it requires fewer training data points. Additionally, it enhances generalization, resulting in superior translation performance compared to random selection method.