TokDrift: When LLM Speaks in Subwords but Code Speaks in Grammar
This identifies a hidden obstacle to reliable code understanding and generation for developers and researchers, though it is incremental as it highlights an existing issue rather than proposing a new solution.
The paper tackled the problem of subword tokenizers in code LLMs causing semantically identical code to be tokenized differently due to superficial factors like whitespace, leading to substantial shifts in model behavior across nine models, with even minor formatting changes affecting performance.
Large language models (LLMs) for code rely on subword tokenizers, such as byte-pair encoding (BPE), learned from mixed natural language text and programming language code but driven by statistics rather than grammar. As a result, semantically identical code snippets can be tokenized differently depending on superficial factors such as whitespace or identifier naming. To measure the impact of this misalignment, we introduce TokDrift, a framework that applies semantic-preserving rewrite rules to create code variants differing only in tokenization. Across nine code LLMs, including large ones with over 30B parameters, even minor formatting changes can cause substantial shifts in model behavior. Layer-wise analysis shows that the issue originates in early embeddings, where subword segmentation fails to capture grammar token boundaries. Our findings identify misaligned tokenization as a hidden obstacle to reliable code understanding and generation, highlighting the need for grammar-aware tokenization for future code LLMs.