Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens
For researchers and practitioners deploying LLMs for reasoning tasks, this work offers a simple, model-agnostic method to reduce inference-time compute while maintaining performance, and provides interpretable annotations of reasoning moves.
The paper identifies two functional types of reasoning tokens in LLMs—low-entropy structural and high-entropy organic—and proposes a compression pipeline using cross-word BPE merges to create supertokens. The approach shortens reasoning traces by 8.1% on average with no significant accuracy loss across three model families and five benchmarks.
Reasoning in Large Language Models incurs significant inference-time compute, yet the token-level information structure of reasoning traces remains underexplored. We observe that reasoning tokens split into two functional types: low-entropy \textit{structural} tokens (recurring phrases that scaffold the reasoning process) and higher-entropy \textit{organic} tokens (problem-specific content that drives toward a solution). This asymmetry motivates a simple, model-agnostic compression pipeline: apply cross-word BPE merges on a model's own reasoning traces to derive \textit{supertokens} that capture frequent structural patterns, then teach the model to adopt them via supervised fine-tuning. Across three model families and five mathematical reasoning benchmarks, our approach shortens reasoning traces by 8.1\% on average with no statistically significant accuracy loss on any model--benchmark pair. Beyond compression, supertokens act as interpretable reasoning-move annotations (backtracking, verification, strategy shifts), exposing the model's high-level strategy at a glance. Analyzing transitions between structural categories reveals systematic differences between correct and incorrect traces: correct traces show productive recovery (backtracking followed by strategy shifts and verification), while incorrect traces are dominated by confusion cycles (repeated hedging and unresolved contradictions). These diagnostic signals suggest applications in reward shaping and early stopping for RL-based reasoning training.