Beyond Text Compression: Evaluating Tokenizers Across Scales
This work addresses the problem of inefficient tokenizer evaluation for language model developers, offering a more reliable framework, though it is incremental in improving existing evaluation methods.
The paper tackled the challenge of evaluating tokenizer quality efficiently, showing that smaller models can predict tokenizer impact on larger models with reduced compute, and found that tokenizer choice has negligible effects in English but consistent differences in multilingual settings, proposing new intrinsic metrics that correlate better with downstream performance.
The choice of tokenizer can profoundly impact language model performance, yet accessible and reliable evaluations of tokenizer quality remain an open challenge. Inspired by scaling consistency, we show that smaller models can accurately predict significant differences in tokenizer impact on larger models at a fraction of the compute cost. By systematically evaluating both English-centric and multilingual tokenizers, we find that tokenizer choice has negligible effects on tasks in English but results in consistent performance differences in multilingual settings. We propose new intrinsic tokenizer metrics inspired by Zipf's law that correlate more strongly with downstream performance than text compression when modeling unseen languages. By combining several metrics to capture multiple aspects of tokenizer behavior, we develop a reliable framework for intrinsic tokenizer evaluations. Our work offers a more efficient path to informed tokenizer selection in future language model development.