False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models
This addresses a key design question for multilingual NLP systems, showing that shared vocabulary is beneficial, though it is incremental in clarifying prior mixed evidence.
The study investigated whether overlapping tokens in multilingual language models facilitate cross-lingual transfer or cause interference, finding that models with overlapping vocabularies outperformed those with disjoint ones on XNLI and XQuAD, with transfer improving as overlap increased.
Subword tokenizers trained on multilingual corpora naturally produce overlapping tokens across languages. Does token overlap facilitate cross-lingual transfer or instead introduce interference between languages? Prior work offers mixed evidence, partly due to varied setups and confounders, such as token frequency or subword segmentation granularity. To address this question, we devise a controlled experiment where we train bilingual autoregressive models on multiple language pairs under systematically varied vocabulary overlap settings. Crucially, we explore a new dimension to understanding how overlap affects transfer: the semantic similarity of tokens shared across languages. We first analyze our models' hidden representations and find that overlap of any kind creates embedding spaces that capture cross-lingual semantic relationships, while this effect is much weaker in models with disjoint vocabularies. On XNLI and XQuAD, we find that models with overlap outperform models with disjoint vocabularies, and that transfer performance generally improves as overlap increases. Overall, our findings highlight the advantages of token overlap in multilingual models and show that substantial shared vocabulary remains a beneficial design choice for multilingual tokenizers.