CAPO: Confidence Aware Preference Optimization Learning for Multilingual Preferences
This addresses the challenge of aligning large language models with human preferences in multilingual contexts, representing an incremental improvement over existing methods like DPO.
The paper tackled the problem of preference optimization methods failing to generalize to multilingual settings by proposing CAPO, a dynamic loss scaling mechanism based on relative reward, which outperformed baselines by at least 16% in reward accuracy and improved alignment across languages.
Preference optimization is a critical post-training technique used to align large language models (LLMs) with human preferences, typically by fine-tuning on ranked response pairs. While methods like Direct Preference Optimization (DPO) have proven effective in English, they often fail to generalize robustly to multilingual settings. We propose a simple yet effective alternative, Confidence-Aware Preference Optimization (CAPO), which replaces DPO's fixed treatment of preference pairs with a dynamic loss scaling mechanism based on a relative reward. By modulating the learning signal according to the confidence in each preference pair, CAPO enhances robustness to noisy or low-margin comparisons, typically encountered in multilingual text. Empirically, CAPO outperforms existing preference optimization baselines by at least 16% in reward accuracy, and improves alignment by widening the gap between preferred and dispreferred responses across languages.