Reference-Free Reinforcement Learning Fine-Tuning for MT: A Seq2Seq Perspective
This work addresses the need for effective fine-tuning of encoder-decoder MT models in low-resource settings where parallel data is scarce, offering a practical alternative to supervised fine-tuning.
The paper applies Group Relative Policy Optimization (GRPO) to encoder-decoder MT models (NLLB-200) using a reference-free reward, achieving consistent improvements across 13 languages (up to +5.03 chrF++ for Traditional Chinese) and matching supervised fine-tuning on morphologically complex languages without any parallel data.
Production machine translation relies overwhelmingly on encoder-decoder Seq2Seq models, yet reinforcement learning approaches to MT fine-tuning have largely targeted decoder-only LLMs at $\geq$7B parameters, with limited systematic study of encoder-decoder architectures. We apply Group Relative Policy Optimization to NLLB-200 (600M and 1.3B) using a hybrid reference-free reward (LaBSE and COMET-Kiwi) that requires no parallel data at fine-tuning time, evaluating across 13 typologically diverse languages. GRPO yields consistent improvements on all 13 languages, up to $+$5.03 chrF++ for Traditional Chinese, and, without any target-language data, competes with 3-epoch supervised fine-tuning on morphologically complex languages . We identify a consistent empirical pattern in which gains are largest where baseline performance is weakest and reward discriminability is highest, making this approach most effective precisely where parallel data is scarcest, and replicate this pattern across English and Spanish source languages.