Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models?
This addresses the performance gap in factual knowledge probing for low-resource languages, offering a method to enhance cross-lingual knowledge transfer in multilingual models.
The paper tackles the limited factual knowledge transfer from high-resource to low-resource languages in multilingual language models by proposing parameter-free language representation projection modules, which significantly improve factual knowledge retrieval accuracy on the mLAMA dataset.
Multilingual pretrained language models serve as repositories of multilingual factual knowledge. Nevertheless, a substantial performance gap of factual knowledge probing exists between high-resource languages and low-resource languages, suggesting limited implicit factual knowledge transfer across languages in multilingual pretrained language models. This paper investigates the feasibility of explicitly transferring relatively rich factual knowledge from English to non-English languages. To accomplish this, we propose two parameter-free $\textbf{L}$anguage $\textbf{R}$epresentation $\textbf{P}$rojection modules (LRP2). The first module converts non-English representations into English-like equivalents, while the second module reverts English-like representations back into representations of the corresponding non-English language. Experimental results on the mLAMA dataset demonstrate that LRP2 significantly improves factual knowledge retrieval accuracy and facilitates knowledge transferability across diverse non-English languages. We further investigate the working mechanism of LRP2 from the perspectives of representation space and cross-lingual knowledge neuron.