Batching BPE Tokenization Merges
This work addresses the challenge of making tokenization experimentation more accessible in compute- and memory-constrained contexts, though it is incremental in nature.
The paper tackles the problem of efficiently building tokenizer vocabularies by batching Byte Pair Encoding merges and reducing memory usage, enabling high-quality tokenizer training on a basic laptop. It presents BatchBPE, an open-source implementation that demonstrates feasibility through training several vocabularies and evaluating encoded text lengths.
The Byte Pair Encoding algorithm can be safely batched to merge hundreds of pairs of tokens at a time when building up a tokenizer's vocabulary. This technique combined with reducing the memory footprint of text used in vocabulary training make it feasible to train a high quality tokenizer on a basic laptop. This paper presents BatchBPE, an open-source pure Python implementation of these concepts, with the goal of making experimenting with new tokenization strategies more accessible especially in compute- and memory-constrained contexts. BatchBPE's usefulness and malleability are demonstrated through the training of several token vocabularies to explore the batch merging process and experiment with preprocessing a stop word list and ignoring the least common text chunks in a dataset. Resultant encoded lengths of texts are used as a basic evaluation metric.