Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural Networks
This work addresses the problem of understanding cross-lingual transfer mechanisms in multilingual models for researchers in NLP and linguistics, but it is incremental as it builds on established multilingual training methods.
The paper investigates whether multilingual training enables sharing of grammatical knowledge beyond lexical alignment, finding that exposure to a related language does not always improve grammatical knowledge in the target language and that conditions optimal for lexical-semantic transfer differ from those for syntactic transfer.
It is now established that modern neural language models can be successfully trained on multiple languages simultaneously without changes to the underlying architecture. But what kind of knowledge is really shared among languages within these models? Does multilingual training mostly lead to an alignment of the lexical representation spaces or does it also enable the sharing of purely grammatical knowledge? In this paper we dissect different forms of cross-lingual transfer and look for its most determining factors, using a variety of models and probing tasks. We find that exposing our LMs to a related language does not always increase grammatical knowledge in the target language, and that optimal conditions for lexical-semantic transfer may not be optimal for syntactic transfer.