Looking for Clues of Language in Multilingual BERT to Improve Cross-lingual Generalization
This addresses the challenge of cross-lingual transfer in NLP by enhancing multilingual models, though it is incremental as it builds on existing m-BERT capabilities.
The study tackled the problem of separating language-specific from semantic information in multilingual BERT embeddings, finding that averaging token embeddings yields language representations and enabling unsupervised token translation, which improved cross-lingual generalization with a computationally cheap method.
Token embeddings in multilingual BERT (m-BERT) contain both language and semantic information. We find that the representation of a language can be obtained by simply averaging the embeddings of the tokens of the language. Given this language representation, we control the output languages of multilingual BERT by manipulating the token embeddings, thus achieving unsupervised token translation. We further propose a computationally cheap but effective approach to improve the cross-lingual ability of m-BERT based on this observation.