Bit-level BPE: Below the byte boundary
This addresses efficiency issues in large language models for character-diverse contexts, but appears incremental as it builds on existing byte-level fallback practices.
The paper tackles the problem of increased sequence length from byte-level tokenization in languages like Chinese, Japanese, and Korean by proposing a lossless compression technique to reduce computation during training and inference.
Byte-level fallbacks for subword tokenization have become a common practice in large language models. In particular, it has been demonstrated to be incredibly effective as a pragmatic solution for preventing OOV, especially in the context of larger models. However, breaking a character down to individual bytes significantly increases the sequence length for long-tail tokens in languages such as Chinese, Japanese, and Korean (CJK) and other character-diverse contexts such as emoji. The increased sequence length results in longer computation during both training and inference. In this work, we propose a simple compression technique that reduces the sequence length losslessly.