Multilingual Fine-Tuning via Localized Gradient Conflict Resolution
For practitioners of multilingual LLM fine-tuning, this work provides a scalable method to mitigate cross-lingual interference, improving performance without prohibitive communication overhead.
The paper addresses negative interference across languages during multilingual fine-tuning of LLMs by reformulating it as a multi-objective optimization problem and introducing Bucket-Level MOO, a distributed framework that applies gradient-based MOO locally on parameter buckets. The method improves both seen and unseen multilingual performance over standard fine-tuning across four base LLMs.
The rapid evolution of Large Language Models (LLMs) has established cross-lingual versatility as a defining feature of modern systems. However, fine-tuning these models frequently induces negative interference across languages. To address this, we reformulate multilingual fine-tuning as a multi-objective optimization (MOO) problem. Specifically, we introduce Bucket-Level MOO, a scalable distributed framework that applies gradient-based MOO algorithms locally on parameter buckets. This enables conflict-aware updates without the prohibitive communication overhead of reconstructing full gradient vectors. Theoretically, we prove this localized resolution natively enforces Refined Pareto Stationarity, a strictly tighter necessary condition for Pareto optimality. Empirically, Bucket-Level MOO mitigates interference by driving LLMs to construct distinct language-specific dimensions, improving representational separability. Extensive experiments across four base LLMs demonstrate that our method significantly improves both seen and unseen multilingual performance over standard fine-tuning paradigms.