Locating Language-Specific Information in Contextualized Embeddings
This work provides insights into MPLM representations for researchers in multilingual NLP, though it is incremental as it builds on existing analysis methods without introducing new models or broad applications.
The study tackled the problem of understanding whether multilingual pretrained language model (MPLM) representations are language-agnostic or interleaved with task-specific heads by locating language-specific information in these models, showing that such information is scattered across dimensions and can be projected into a linear subspace.
Multilingual pretrained language models (MPLMs) exhibit multilinguality and are well suited for transfer across languages. Most MPLMs are trained in an unsupervised fashion and the relationship between their objective and multilinguality is unclear. More specifically, the question whether MPLM representations are language-agnostic or they simply interleave well with learned task prediction heads arises. In this work, we locate language-specific information in MPLMs and identify its dimensionality and the layers where this information occurs. We show that language-specific information is scattered across many dimensions, which can be projected into a linear subspace. Our study contributes to a better understanding of MPLM representations, going beyond treating them as unanalyzable blobs of information.