Meta-Learning Preferences for Multilingual LLM Alignment
For researchers and practitioners working on multilingual LLM alignment, this work provides a method to effectively adapt to low-resource languages with minimal data, addressing a critical bottleneck in multilingual AI.
The paper tackles the challenge of aligning large language models in multilingual settings where human preference data is scarce for low-resource languages. It proposes a meta-learning framework that achieves up to 28% win-rate improvements over baselines with only 100 target-language preference samples.
Unequal availability of human preference data across languages poses a significant challenge for aligning large language models in multilingual settings. To address the lack of sufficient data in low-resource language alignment, we propose a meta-learning framework for Reinforcement Learning from Human Feedback and Direct Preference Optimization. By leveraging preference data from other languages, our framework learns a transferable initialization that enables effective adaptation to a target language with minimal data. We provide theoretical guarantees for both the meta-reward modeling and meta-policy optimization settings, and empirically demonstrate the effectiveness of our approach on multilingual benchmarks. In an extremely low-resource setting with only 100 target-language preference samples, our approach achieves up to $28\%$ win-rate improvements over baseline methods, and consistently outperforms baselines across multiple target languages and model scales. Our approaches retain these advantages across different combinations of meta-training languages and varying linguistic distances from the target languages.