Understanding the role of FFNs in driving multilingual behaviour in LLMs
This work addresses the under-explored problem of multilingualism in LLMs for researchers and practitioners, though it is incremental as it builds on existing architectures with new analysis.
The paper investigates how Feed-Forward Networks (FFNs) influence multilingual behavior in Large Language Models, revealing distinct processing patterns in FFN sublayers and identifying 'over-layerization' as a factor that can degrade performance with increased depth.
Multilingualism in Large Language Models (LLMs) is an yet under-explored area. In this paper, we conduct an in-depth analysis of the multilingual capabilities of a family of a Large Language Model, examining its architecture, activation patterns, and processing mechanisms across languages. We introduce novel metrics to probe the model's multilingual behaviour at different layers and shed light on the impact of architectural choices on multilingual processing. Our findings reveal different patterns of multilinugal processing in the sublayers of Feed-Forward Networks of the models. Furthermore, we uncover the phenomenon of "over-layerization" in certain model configurations, where increasing layer depth without corresponding adjustments to other parameters may degrade model performance. Through comparisons within and across languages, we demonstrate the interplay between model architecture, layer depth, and multilingual processing capabilities of LLMs trained on multiple languages.