BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training
This addresses tokenization inefficiencies for language model developers, but it is incremental as it builds on the classical BPE method.
The paper tackles the problem of sub-optimal tokenization in language models by introducing Picky BPE, a modified BPE algorithm that refines vocabulary during training to improve efficiency and eliminate under-trained tokens, resulting in maintained or improved downstream performance in several cases.
Language models can largely benefit from efficient tokenization. However, they still mostly utilize the classical BPE algorithm, a simple and reliable method. This has been shown to cause such issues as under-trained tokens and sub-optimal compression that may affect the downstream performance. We introduce Picky BPE, a modified BPE algorithm that carries out vocabulary refinement during tokenizer training. Our method improves vocabulary efficiency, eliminates under-trained tokens, and does not compromise text compression. Our experiments show that our method does not reduce the downstream performance, and in several cases improves it.