Cross-lingual Human-Preference Alignment for Neural Machine Translation with Direct Quality Optimization
This addresses the problem of aligning machine translation outputs with human preferences for users of multilingual translation systems, though it is incremental as it adapts existing RLHF/DPO methods to a specific domain.
The paper tackled the task-data mismatch in neural machine translation by introducing Direct Quality Optimization (DQO), a variant of DPO that uses a pre-trained quality estimation model as a proxy for human preferences, resulting in improvements across all languages of a multilingual model, even when applied only to a subset, as verified by automatic metrics and human evaluation.
Reinforcement Learning from Human Feedback (RLHF) and derivative techniques like Direct Preference Optimization (DPO) are task-alignment algorithms used to repurpose general, foundational models for specific tasks. We show that applying task-alignment to neural machine translation (NMT) addresses an existing task--data mismatch in NMT, leading to improvements across all languages of a multilingual model, even when task-alignment is only applied to a subset of those languages. We do so by introducing Direct Quality Optimization (DQO), a variant of DPO leveraging a pre-trained translation quality estimation model as a proxy for human preferences, and verify the improvements with both automatic metrics and human evaluation.