Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models
This addresses the foundational problem of tokenization design for developers and researchers in natural language processing, though it is incremental as it builds on existing subword methods.
The paper tackles the problem of tokenization being an under-theorized component in large language models, which leads to issues like misalignment with linguistic structure and bias amplification, and argues for reframing it as a core design decision to improve fairness, efficiency, and adaptability.
Tokenization underlies every large language model, yet it remains an under-theorized and inconsistently designed component. Common subword approaches such as Byte Pair Encoding (BPE) offer scalability but often misalign with linguistic structure, amplify bias, and waste capacity across languages and domains. This paper reframes tokenization as a core modeling decision rather than a preprocessing step. We argue for a context-aware framework that integrates tokenizer and model co-design, guided by linguistic, domain, and deployment considerations. Standardized evaluation and transparent reporting are essential to make tokenization choices accountable and comparable. Treating tokenization as a core design problem, not a technical afterthought, can yield language technologies that are fairer, more efficient, and more adaptable.