LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers
This addresses a specific issue in tokenization for language models, offering an incremental improvement by optimizing vocabulary efficiency and robustness.
The paper tackled the problem of intermediate merge residues in BPE tokenizers, which waste vocabulary capacity and increase vulnerability to adversarial inputs, and introduced LiteToken to remove these tokens, resulting in reduced token fragmentation, fewer parameters, and improved robustness to noisy inputs while maintaining performance.
Tokenization is fundamental to how language models represent and process text, yet the behavior of widely used BPE tokenizers has received far less study than model architectures and training. In this paper, we investigate intermediate merge residues in BPE vocabularies: tokens that are frequent during merge learning so that retained in the final vocabulary, but are mostly further merged and rarely emitted when tokenizing the corpus during tokenizer usage. Such low-frequency tokens not only waste vocabulary capacity but also increase vulnerability to adversarial or atypical inputs. We present a systematic empirical characterization of this phenomenon across commonly used tokenizers and introduce LiteToken, a simple method for removing residue tokens. Because the affected tokens are rarely used, pretrained models can often accommodate the modified tokenizer without additional fine-tuning. Experiments show that LiteToken reduces token fragmentation, reduces parameters, and improves robustness to noisy or misspelled inputs, while preserving overall performance.