Tokenization is Sensitive to Language Variation
This addresses the problem of tokenizer sensitivity to language variation for NLP researchers and practitioners, offering insights for optimizing LLM performance on diverse tasks, though it is incremental in refining existing tokenization methods.
The study investigated how tokenizer design choices, such as corpus, pre-tokenizer, and vocabulary size, affect downstream LLM performance on tasks requiring robustness or sensitivity to language variation, finding that the best tokenizer varies by task type and the pre-tokenizer has the biggest impact, with a new estimation method showing substantial improvement over existing metrics.
Variation in language is ubiquitous and often systematically linked to regional, social, and contextual factors. Tokenizers split texts into smaller units and might behave differently for less common linguistic forms. This might affect downstream LLM performance differently on two types of tasks: Tasks where the model should be robust to language variation (e.g., for semantic tasks like NLI, labels do not depend on whether a text uses British or American spelling) and tasks where the model should be sensitive to language variation (e.g., for form-based tasks like authorship verification, labels depend on whether a text uses British or American spelling). We pre-train BERT base models with the popular Byte-Pair Encoding algorithm to investigate how key tokenization design choices impact the performance of downstream models: the corpus used to train the tokenizer, the pre-tokenizer and the vocabulary size. We find that the best tokenizer varies on the two task types and that the pre-tokenizer has the biggest overall impact on performance. Further, we introduce a new approach to estimate tokenizer impact on downstream LLM performance, showing substantial improvement over metrics like Rényi efficiency. We encourage more work on language variation and its relation to tokenizers and thus LLM performance.