Which Pieces Does Unigram Tokenization Really Need?
This work addresses a practical bottleneck for researchers and practitioners using tokenization methods, offering incremental improvements to enhance usability and efficiency.
The paper tackles the complexity of implementing the Unigram tokenization algorithm by providing a clear implementation guide and parameter choices, and introduces a simpler algorithm that achieves improved compression at the cost of slightly higher training loss.
The Unigram tokenization algorithm offers a probabilistic alternative to the greedy heuristics of Byte-Pair Encoding. Despite its theoretical elegance, its implementation in practice is complex, limiting its adoption to the SentencePiece package and adapters thereof. We bridge this gap between theory and practice by providing a clear guide to implementation and parameter choices. We also identify a simpler algorithm that accepts slightly higher training loss in exchange for improved compression.