Broken Words, Broken Performance: Effect of Tokenization on Performance of LLMs
This addresses a fundamental preprocessing issue in LLMs that could affect performance for NLP researchers and practitioners, though it is incremental as it builds on known tokenization challenges.
The paper investigates how tokenization in Large Language Models (LLMs), which can break natural words into multiple tokens, negatively impacts performance on NLP tasks, and proposes penalty functions to quantify this effect, showing statistical significance across multiple models and tasks.
Tokenization is the first step in training any Large Language Model (LLM), where the text is split into a sequence of tokens as per the model's fixed vocabulary. This tokenization in LLMs is different from the traditional tokenization in NLP where the text is split into a sequence of "natural" words. In LLMs, a natural word may also be broken into multiple tokens due to limited vocabulary size of the LLMs (e.g., Mistral's tokenizer splits "martial" into "mart" and "ial"). In this paper, we hypothesize that such breaking of natural words negatively impacts LLM performance on various NLP tasks. To quantify this effect, we propose a set of penalty functions that compute a tokenization penalty for a given text for a specific LLM, indicating how "bad" the tokenization is. We establish statistical significance of our hypothesis on multiple NLP tasks for a set of different LLMs.