Parameter Alignment Mitigates Catastrophic Forgetting in Multilingual Expert Language Models
For practitioners deploying multilingual expert models, this work provides practical guidelines to mitigate forgetting during continual pretraining.
The paper addresses catastrophic forgetting in multilingual continual pretraining of large language models and proposes five parameter alignment strategies. These strategies reduce forgetting with minimal cost to language acquisition, with layer freezing and regularization best preserving comprehension and post-hoc reversion yielding strongest translation gains.
While continual pretraining~(CPT) is a practical way to extend large language models to new languages, naïve finetuning on targeted data erodes existing capabilities through catastrophic forgetting. Organizing training around language families reduces cross-language interference but cannot alone prevent forgetting of the general knowledge needed for downstream tasks. We link this forgetting to parameter drift in multilingual CPT and present a suite of five layer-aware parameter alignment strategies: hard layer freezing, soft regularization, post-hoc weight reversion, and model merging. We systematically compare our alignment strategies against two unregularized CPT baselines on benchmarks spanning 32 training languages from five language families, plus held-out languages, across four evaluation axes: perplexity, reading comprehension, physical reasoning, and translation. Parameter alignment substantially reduces forgetting at minimal cost to language acquisition: layer freezing and regularization best preserve comprehension, whereas post-hoc reversion yields the strongest translation gains. Together, these results map the acquisition--forgetting frontier for family-expert CPT and offer practical deployment guidelines pairing each strategy to the tasks it best serves.