Do Multilingual LLMs have specialized language heads?
This addresses the problem of inefficient deployment for multilingual LLMs when only a subset of languages is needed, offering a potential solution for practitioners, though it appears incremental as it builds on prior work in machine translation models.
This paper investigates whether multilingual large language models have specialized attention heads for each language and explores removing heads for unwanted languages to maintain performance in targeted languages, finding that this can enable more efficient deployment with reduced complexity.
Multilingual large language models (LLMs) have gained significant popularity for their ability to process and generate text across multiple languages. However, deploying these models in production can be inefficient when only a subset of the supported languages is of interest. There has been some research conducted on identifying whether machine translation models have language-specific or language-agnostic heads, however no research has been conducted for multilingual LLMs, to the best of our knowledge, that as we know are capable of performing diverse tasks beyond just translation. This paper explores whether multilingual LLMs have specialized language attention heads for each language, and investigates the possibility of removing language-specific heads for unwanted languages without degrading performance in the targeted languages. Our findings could inform more efficient deployment strategies for multilingual LLMs, enabling reduced model complexity while maintaining high accuracy for targeted languages.