From Characters to Tokens: Dynamic Grouping with Hierarchical BPE
This addresses tokenization bottlenecks for NLP researchers and practitioners, offering a language-agnostic solution, though it is incremental as it builds on hierarchical BPE methods.
The paper tackles inefficiencies in subword tokenization for large language models, such as representing rare words and large embedding matrices, by proposing a dynamic character grouping method that leverages BPE structure without extra models, achieving performance matching or exceeding existing patching strategies with a compact vocabulary.
Subword tokenization methods like Byte Pair Encoding (BPE) are widely used in large language models due to their balance of vocabulary compactness and representational power. However, they suffer from inefficiencies in representing rare words and require large embedding matrices. Character-level models address these issues but introduce performance bottlenecks, particularly in Transformer-based architectures. Recent hierarchical models attempt to merge the benefits of both paradigms by grouping characters into patches, but existing patching strategies either rely on whitespace-limiting applicability to certain languages, or require auxiliary models that introduce new dependencies. In this paper, we propose a dynamic character grouping method that leverages the structure of existing BPE tokenization without requiring additional models. By appending explicit end-of-patch markers to BPE tokens and introducing a second-level BPE compression stage to control patch granularity, our method offers efficient, flexible, and language-agnostic representations. Empirical results demonstrate that our approach matches or exceeds the performance of dynamic entropy- and whitespace-based patching strategies, while maintaining a compact vocabulary.