Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders
This work provides a method for analyzing and controlling language-specific behaviors in multilingual LLMs, which is incremental but offers practical tools for researchers and developers.
The researchers tackled the challenge of understanding multilingual capabilities in Large Language Models by using Sparse Autoencoders to identify language-specific features, discovering that ablating these features reduces abilities in only one language while leaving others unaffected, and leveraging them to enhance steering vectors for language control.
The mechanisms behind multilingual capabilities in Large Language Models (LLMs) have been examined using neuron-based or internal-activation-based methods. However, these methods often face challenges such as superposition and layer-wise activation variance, which limit their reliability. Sparse Autoencoders (SAEs) offer a more nuanced analysis by decomposing the activations of LLMs into a sparse linear combination of SAE features. We introduce a novel metric to assess the monolinguality of features obtained from SAEs, discovering that some features are strongly related to specific languages. Additionally, we show that ablating these SAE features only significantly reduces abilities in one language of LLMs, leaving others almost unaffected. Interestingly, we find some languages have multiple synergistic SAE features, and ablating them together yields greater improvement than ablating individually. Moreover, we leverage these SAE-derived language-specific features to enhance steering vectors, achieving control over the language generated by LLMs. The code is publicly available at https://github.com/Aatrox103/multilingual-llm-features.