Parallel Tokenizers: Rethinking Vocabulary Design for Cross-Lingual Transfer
This addresses the challenge of cross-lingual generalization for low-resource languages, though it is an incremental improvement over existing tokenization methods.
The paper tackled the problem of ineffective cross-lingual transfer in multilingual language models by introducing parallel tokenizers, which align vocabularies across languages to ensure consistent indices for semantically equivalent words, resulting in models that outperform conventional baselines across tasks like sentiment analysis and hate speech detection.
Tokenization defines the foundation of multilingual language models by determining how words are represented and shared across languages. However, existing methods often fail to support effective cross-lingual transfer because semantically equivalent words are assigned distinct embeddings. For example, "I eat rice" in English and "Ina cin shinkafa" in Hausa are typically mapped to different vocabulary indices, preventing shared representations and limiting cross-lingual generalization. We introduce parallel tokenizers. This new framework trains tokenizers monolingually and then aligns their vocabularies exhaustively using bilingual dictionaries or word-to-word translation, ensuring consistent indices for semantically equivalent words. This alignment enforces a shared semantic space across languages while naturally improving fertility balance. To assess their effectiveness, we pretrain a transformer encoder from scratch on thirteen low-resource languages and evaluate it on sentiment analysis, hate speech detection, emotion classification, and sentence embedding similarity. Across all tasks, models trained with parallel tokenizers outperform conventional multilingual baselines, confirming that rethinking tokenization is essential for advancing multilingual representation learning--especially in low-resource settings.