CharBench: Evaluating the Role of Tokenization in Character-Level Tasks
This addresses the problem of evaluating and improving language models' performance on character-level reasoning for researchers and developers, though it is incremental as it builds on existing work with a new benchmark.
The paper tackled the unclear impact of tokenization on character-level tasks by introducing CharBench, a large benchmark, and found that modern LLMs struggle with an average accuracy of 43.6% and 32.3% on some tasks, with tokenization properties weakly correlated for counting but negatively correlated for positional tasks.
Tasks that require character-level reasoning, such as counting or locating characters within words, remain challenging for contemporary language models. A common conjecture is that language models' reliance on subword units, rather than characters, contributes to their struggles with character-level tasks, yet recent studies offer conflicting conclusions about the role of tokenization, leaving its impact unclear. To address this gap, we introduce CharBench, a comprehensive benchmark of character-level tasks that is two orders of magnitude larger than existing alternatives. We evaluate a diverse range of leading open-weight and proprietary models on CharBench and find that it presents a significant challenge to modern LLMs, with an average accuracy of 43.6% and 32.3% on some tasks. We present an in-depth analysis of how intrinsic properties of words and their segmentations into tokens correspond to model performance. For counting tasks, we find that tokenization properties are weakly correlated with correctness, while the length of the queried word and the actual character count play a more significant part. In contrast, for tasks requiring intra-word positional understanding, performance is negatively correlated with the length of the token containing the queried character, suggesting that longer tokens obscure character position information for LLMs. We encourage future work to build on the benchmark and evaluation methodology introduced here as tools for improving model performance on such tasks.