Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet
This work addresses performance disparities in multilingual language models caused by tokenization biases, offering a more equitable representation for low-resource and non-Latin script languages.
The paper proposes using the International Phonetic Alphabet (IPA) as a language-agnostic input representation for multilingual tokenizers, showing that IPA tokenizers consistently improve tokenization quality across 24 languages and 14 scripts, especially for non-Latin scripts, and generalize better to unseen languages and scripts.
Multilingual language models often exhibit performance disparities across languages that can arise as early as the tokenization stage. Widely-used subword tokenization approaches favor high-resource languages, and tokenizer-free methods still yield longer sequences for scripts with a higher bytes-per-character ratio. To address these shortcomings, we propose to use the International Phonetic Alphabet (IPA) as a language-agnostic input representation for multilingual tokenizers. IPA provides a compact symbol inventory, greater cross-lingual character overlap, and a more balanced byte-per-character distribution across languages. We train matched pairs of text vs. IPA subword tokenizers across 24 languages and 14 scripts and demonstrate that IPA tokenizers consistently improve tokenization quality, especially for non-Latin scripts, and generalize more effectively to unseen languages and scripts.