LBPE: Long-token-first Tokenization to Improve Large Language Models
This addresses a specific bottleneck in tokenization for LLMs, offering an incremental improvement to enhance model learning efficiency.
The paper tackles the problem of imbalanced learning in Large Language Models due to long tokens having fewer occurrences than short tokens in Byte Pair Encoding tokenization, and proposes LBPE which prioritizes long tokens to smooth frequency differences, resulting in consistent performance improvements over BPE across diverse tasks.
The prevalent use of Byte Pair Encoding (BPE) in Large Language Models (LLMs) facilitates robust handling of subword units and avoids issues of out-of-vocabulary words. Despite its success, a critical challenge persists: long tokens, rich in semantic information, have fewer occurrences in tokenized datasets compared to short tokens, which can result in imbalanced learning issue across different tokens. To address that, we propose LBPE, which prioritizes long tokens during the encoding process. LBPE generates tokens according to their reverse ranks of token length rather than their ranks in the vocabulary, granting longer tokens higher priority during the encoding process. Consequently, LBPE smooths the frequency differences between short and long tokens, and thus mitigates the learning imbalance. Extensive experiments across diverse language modeling tasks demonstrate that LBPE consistently outperforms the original BPE, well demonstrating its effectiveness.