Word Recovery in Large Language Models Enables Character-Level Tokenization Robustness
This provides a mechanistic explanation for tokenization robustness in LLMs, which is incremental as it clarifies an existing phenomenon rather than introducing a new method.
The study tackled the problem of understanding why large language models (LLMs) are robust to non-canonical inputs like character-level tokenization, and identified a core mechanism called word recovery, where hidden states reconstruct canonical word-level tokens, with causal evidence showing its removal degrades task performance.
Large language models (LLMs) trained with canonical tokenization exhibit surprising robustness to non-canonical inputs such as character-level tokenization, yet the mechanisms underlying this robustness remain unclear. We study this phenomenon through mechanistic interpretability and identify a core process we term word recovery. We first introduce a decoding-based method to detect word recovery, showing that hidden states reconstruct canonical word-level token identities from character-level inputs. We then provide causal evidence by removing the corresponding subspace from hidden states, which consistently degrades downstream task performance. Finally, we conduct a fine-grained attention analysis and show that in-group attention among characters belonging to the same canonical token is critical for word recovery: masking such attention in early layers substantially reduces both recovery scores and task performance. Together, our findings provide a mechanistic explanation for tokenization robustness and identify word recovery as a key mechanism enabling LLMs to process character-level inputs.