LayAlign: Enhancing Multilingual Reasoning in Large Language Models via Layer-Wise Adaptive Fusion and Alignment Strategy
This addresses the issue of multilingual reasoning for users of low-resource languages, representing an incremental improvement over prior methods.
The paper tackles the problem of suboptimal performance of large language models on low-resource languages by proposing a framework that integrates representations from all layers of a multilingual encoder, resulting in consistent outperformance over existing baselines on multilingual reasoning tasks.
Despite being pretrained on multilingual corpora, large language models (LLMs) exhibit suboptimal performance on low-resource languages. Recent approaches have leveraged multilingual encoders alongside LLMs by introducing trainable parameters connecting the two models. However, these methods typically focus on the encoder's output, overlooking valuable information from other layers. We propose \aname (\mname), a framework that integrates representations from all encoder layers, coupled with the \attaname mechanism to enable layer-wise interaction between the LLM and the multilingual encoder. Extensive experiments on multilingual reasoning tasks, along with analyses of learned representations, show that our approach consistently outperforms existing baselines.