Positional Cognitive Specialization: Where Do LLMs Learn To Comprehend and Speak Your Language?
This work addresses the challenge of efficient multilingual adaptation for LLMs, offering a method to reduce computational costs, though it is incremental as it builds on existing fine-tuning techniques.
The study tackled the problem of expensive and opaque adaptation of large language models to new languages by investigating how they acquire languages during training, and proposed CogSym, a layer-wise heuristic that fine-tunes only 25% of layers to achieve performance within 2-3% deviation from full fine-tuning.
Adapting large language models (LLMs) to new languages is an expensive and opaque process. Understanding how language models acquire new languages and multilingual abilities is key to achieve efficient adaptation. Prior work on multilingual interpretability research focuses primarily on how trained models process multilingual instructions, leaving unexplored the mechanisms through which they acquire new languages during training. We investigate these training dynamics on decoder-only transformers through the lens of two functional cognitive specializations: language perception (input comprehension) and production (output generation). Through experiments on low-resource languages, we demonstrate how perceptual and productive specialization emerges in different regions of a language model by running layer ablation sweeps from the model's input and output directions. Based on the observed specialization patterns, we propose CogSym, a layer-wise heuristic that enables effective adaptation by exclusively fine-tuning a few early and late layers. We show that tuning only the 25% outermost layers achieves downstream task performance within 2-3% deviation from the full fine-tuning baseline. CogSym yields consistent performance with adapter methods such as LoRA, showcasing generalization beyond full fine-tuning. These findings provide insights to better understand how LLMs learn new languages and push toward accessible and inclusive language modeling.