Exploring Language-Agnosticity in Function Vectors: A Case Study in Machine Translation
This work provides evidence for language-agnostic task representations in LLMs, which could simplify cross-lingual transfer, but the findings are incremental as they extend known properties of FVs to multilingual settings.
The paper investigates whether function vectors (FVs) in multilingual LLMs are language-agnostic, using machine translation as a case study. Across three models, translation FVs extracted from English→Target directions transfer to other target languages, improving correct token ranks, and ablation degrades translation across languages.
Function vectors (FVs) are vector representations of tasks extracted from model activations during in-context learning. While prior work has shown that multilingual model representations can be language-agnostic, it remains unclear whether the same holds for function vectors. We study whether FVs exhibit language-agnosticity, using machine translation as a case study. Across three decoder-only multilingual LLMs, we find that translation FVs extracted from a single English$\rightarrow$Target direction transfer to other target languages, consistently improving the rank of correct translation tokens across multiple unseen languages. Ablation results show that removing the FV degrades translation across languages with limited impact on unrelated tasks. We further show that base-model FVs transfer to instruction-tuned variants and partially generalize from word-level to sentence-level translation.