LoPT: Lossless Parallel Tokenization Acceleration for Long Context Inference of Large Language Model
This work solves the problem of slow tokenization in long-text scenarios for users of large language models, representing an incremental improvement over existing parallel methods by eliminating boundary artifacts.
The paper tackles the computational latency in long context inference for large language models by addressing tokenization as a bottleneck, proposing LoPT, a lossless parallel tokenization framework that ensures output identical to sequential tokenization and achieves significant speedup in experiments.
Long context inference scenarios have become increasingly important for large language models, yet they introduce significant computational latency. While prior research has optimized long-sequence inference through operators, model architectures, and system frameworks, tokenization remains an overlooked bottleneck. Existing parallel tokenization methods accelerate processing through text segmentation and multi-process tokenization, but they suffer from inconsistent results due to boundary artifacts that occur after merging. To address this, we propose LoPT, a novel Lossless Parallel Tokenization framework that ensures output identical to standard sequential tokenization. Our approach employs character-position-based matching and dynamic chunk length adjustment to align and merge tokenized segments accurately. Extensive experiments across diverse long-text datasets demonstrate that LoPT achieves significant speedup while guaranteeing lossless tokenization. We also provide theoretical proof of consistency and comprehensive analytical studies to validate the robustness of our method.