On the limited utility of parallel data for learning shared multilingual representations
For researchers in multilingual NLP, this challenges the necessity of parallel data for cross-lingual representation learning, suggesting alternative approaches may suffice.
This study investigates the role of parallel data in pretraining multilingual models and finds that parallel data has minimal effect on cross-lingual alignment, with alignment emerging similarly even without it. The effect is limited to accelerating early-phase sharing and reducing language-specific neurons.
Shared multilingual representations are essential for cross-lingual tasks and knowledge transfer across languages. This study looks at the impact of parallel data, i.e. translated sentences, in pretraining as a signal to trigger representations that are aligned across languages. We train reference models with different proportions of parallel data and show that parallel data seem to have only a minimal effect on the cross-lingual alignment. Based on multiple evaluation methods, we find that the effect is limited to potentially accelerating the representation sharing in the early phases of pretraining, and to decreasing the amount of language-specific neurons in the model. Cross-lingual alignment seems to emerge on similar levels even without the explicit signal from parallel data.