On the Acquisition of Shared Grammatical Representations in Bilingual Language Models
This work addresses the problem of understanding crosslingual transfer mechanisms in language models for researchers in computational linguistics, but it is incremental as it builds on prior structural priming studies.
The study investigated how bilingual language models acquire shared grammatical representations by training small models with controlled language data and exposure order, using structural priming to assess crosslingual transfer. They found asymmetrical priming effects across language pairs and directions, with less robust effects for typologically diverse languages, highlighting limitations in crosslingual transfer.
Crosslingual transfer is crucial to contemporary language models' multilingual capabilities, but how it occurs is not well understood. We ask what happens to a monolingual language model when it begins to be trained on a second language. Specifically, we train small bilingual models for which we control the amount of data for each language and the order of language exposure. To find evidence of shared multilingual representations, we turn to structural priming, a method used to study grammatical representations in humans. We first replicate previous crosslingual structural priming results and find that after controlling for training data quantity and language exposure, there are asymmetrical effects across language pairs and directions. We argue that this asymmetry may shape hypotheses about human structural priming effects. We also find that structural priming effects are less robust for less similar language pairs, highlighting potential limitations of crosslingual transfer learning and shared representations for typologically diverse languages.