Greed is All You Need: An Evaluation of Tokenizer Inference Methods
This work addresses the often unspecified or ill-suited decoding methods in NLP tokenization, providing a controlled analysis for researchers and practitioners.
The paper tackled the problem of evaluating tokenizer inference methods for subword tokenizers in NLP, showing that greedy inference performs well for common tokenizers and that SaGe outperforms others on morphological alignment.
While subword tokenizers such as BPE and WordPiece are typically used to build vocabularies for NLP models, the method of decoding text into a sequence of tokens from these vocabularies is often left unspecified, or ill-suited to the method in which they were constructed. We provide a controlled analysis of seven tokenizer inference methods across four different algorithms and three vocabulary sizes, performed on a novel intrinsic evaluation suite we curated for English, combining measures rooted in morphology, cognition, and information theory. We show that for the most commonly used tokenizers, greedy inference performs surprisingly well; and that SaGe, a recently-introduced contextually-informed tokenizer, outperforms all others on morphological alignment.