Two Counterexamples to Tokenization and the Noiseless Channel
This work addresses a specific issue for NLP researchers and practitioners by highlighting limitations in tokenization evaluation metrics, though it is incremental as it builds on prior work to expose flaws.
The paper tackles the problem of using Rényi efficiency as an intrinsic metric for tokenizer evaluation by presenting two counterexamples where increasing Rényi efficiency arbitrarily decreases downstream model performance, showing its failure as a predictor.
In Tokenization and the Noiseless Channel (Zouhar et al., 2023a), Rényi efficiency is suggested as an intrinsic mechanism for evaluating a tokenizer: for NLP tasks, the tokenizer which leads to the highest Rényi efficiency of the unigram distribution should be chosen. The Rényi efficiency is thus treated as a predictor of downstream performance (e.g., predicting BLEU for a machine translation task), without the expensive step of training multiple models with different tokenizers. Although useful, the predictive power of this metric is not perfect, and the authors note there are additional qualities of a good tokenization scheme that Rényi efficiency alone cannot capture. We describe two variants of BPE tokenization which can arbitrarily increase Rényi efficiency while decreasing the downstream model performance. These counterexamples expose cases where Rényi efficiency fails as an intrinsic tokenization metric and thus give insight for building more accurate predictors.