Learning from others' mistakes: Finetuning machine translation models with span-level error annotations
This work addresses the challenge of leveraging detailed error feedback for machine translation, which is incremental as it builds on existing fine-tuning methods with more granular annotations.
The paper tackled the problem of improving machine translation models by using fine-grained span-level error annotations, and the result was that their TWA algorithm outperformed baselines like Supervised FineTuning and Direct Preference Optimization on English-German and Chinese-English translation tasks.
Despite growing interest in incorporating feedback to improve language models, most efforts focus only on sequence-level annotations. In this work, we explore the potential of utilizing fine-grained span-level annotations from offline datasets to improve model quality. We develop a simple finetuning algorithm, called Training with Annotations (TWA), to directly train machine translation models on such annotated data. TWA utilizes targeted span-level error information while also flexibly learning what to penalize within a span. Moreover, TWA considers the overall trajectory of a sequence when deciding which non-error spans to utilize as positive signals. Experiments on English-German and Chinese-English machine translation show that TWA outperforms baselines such as Supervised FineTuning on sequences filtered for quality and Direct Preference Optimization on pairs constructed from the same data.