Tokenizer Choice For LLM Training: Negligible or Crucial?
This addresses a blind spot in LLM development by showing tokenizer selection is crucial for efficiency and performance, particularly in multilingual settings, though it is incremental as it builds on existing tokenizer methods.
The study investigates the impact of tokenizer choice on large language model performance and training costs, finding that it significantly affects downstream results and that common evaluation metrics are poor predictors, with multilingual tokenizers requiring triple the vocabulary size of English ones and using English-centric tokenizers causing up to 68% higher training costs.
The recent success of Large Language Models (LLMs) has been predominantly driven by curating the training dataset composition, scaling of model architectures and dataset sizes and advancements in pretraining objectives, leaving tokenizer influence as a blind spot. Shedding light on this underexplored area, we conduct a comprehensive study on the influence of tokenizer choice on LLM downstream performance by training 24 mono- and multilingual LLMs at a 2.6B parameter scale, ablating different tokenizer algorithms and parameterizations. Our studies highlight that the tokenizer choice can significantly impact the model's downstream performance and training costs. In particular, we find that the common tokenizer evaluation metrics fertility and parity are not always predictive of model downstream performance, rendering these metrics a questionable proxy for the model's downstream performance. Furthermore, we show that multilingual tokenizers trained on the five most frequent European languages require vocabulary size increases of factor three in comparison to English. While English-centric tokenizers have been applied to the training of multi-lingual LLMs in the past, we find that this approach results in a severe downstream performance degradation and additional training costs of up to 68%, due to an inefficient tokenization vocabulary.