Pre-trained Models Perform the Best When Token Distributions Follow Zipf's Law
This addresses a fundamental bottleneck in NLP and other sequence modeling fields by providing a principled criterion for vocabulary size selection, which is incremental as it builds on known statistical principles.
The paper tackles the problem of selecting optimal vocabulary sizes for tokenization in sequence modeling by proposing a method based on Zipf's law, showing that models achieve peak performance when token distributions follow this law, with consistent improvements across NLP, genomics, and chemistry domains.
Tokenization is a fundamental step in natural language processing (NLP) and other sequence modeling domains, where the choice of vocabulary size significantly impacts model performance. Despite its importance, selecting an optimal vocabulary size remains underexplored, typically relying on heuristics or dataset-specific choices. In this work, we propose a principled method for determining the vocabulary size by analyzing token frequency distributions through Zipf's law. We show that downstream task performance correlates with how closely token distributions follow power-law behavior, and that aligning with Zipfian scaling improves both model efficiency and effectiveness. Extensive experiments across NLP, genomics, and chemistry demonstrate that models consistently achieve peak performance when the token distribution closely adheres to Zipf's law, establishing Zipfian alignment as a robust and generalizable criterion for vocabulary size selection.