Evaluating Morphological Alignment of Tokenizers in 70 Languages
This work addresses the problem of evaluating tokenizer quality for NLP researchers, but it is incremental as it extends an existing metric with limited new insights.
The study expanded MorphScore to evaluate tokenizer morphological alignment across 70 languages and found that this alignment explains little variance in downstream task performance, indicating it is not a key measure of tokenizer quality for model effectiveness.
While tokenization is a key step in language modeling, with effects on model training and performance, it remains unclear how to effectively evaluate tokenizer quality. One proposed dimension of tokenizer quality is the extent to which tokenizers preserve linguistically meaningful subwords, aligning token boundaries with morphological boundaries within a word. We expand MorphScore (Arnett & Bergen, 2025), which previously covered 22 languages, to support a total of 70 languages. The updated MorphScore offers more flexibility in evaluation and addresses some of the limitations of the original version. We then correlate our alignment scores with downstream task performance for five pre-trained languages models on seven tasks, with at least one task in each of the languages in our sample. We find that morphological alignment does not explain very much variance in model performance, suggesting that morphological alignment alone does not measure dimensions of tokenization quality relevant to model performance.