Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluation
This addresses the need for better evaluation metrics in tokenization to improve fairness across languages in NLP applications, though it is incremental as it builds on existing metrics.
The paper tackled the problem of evaluating tokenizers in multilingual LLMs, finding that the standard fertility metric obscures cross-lingual fairness, and proposed STRR to reveal systematic biases, such as prioritization of English and fragmentation in Hindi, offering practical guidance for more equitable tokenizer design.
Tokenization is a crucial but under-evaluated step in large language models (LLMs). The standard metric, fertility (the average number of tokens per word), captures compression efficiency but obscures how vocabularies are allocated across languages and domains. We analyze six widely used tokenizers across seven languages and two domains, finding stable fertility for English, high fertility for Chinese, and little domain sensitivity. To address fertility's blind spots, we propose the Single Token Retention Rate (STRR), which measures the proportion of words preserved as single tokens. STRR reveals systematic prioritization of English, strong support for Chinese, and fragmentation in Hindi, offering an interpretable view of cross-lingual fairness. Our results show that STRR complements fertility and provides practical guidance for designing more equitable multilingual tokenizers.