Behavior-Equivalent Token: Single-Token Replacement for Long Prompts in LLMs
This addresses efficiency issues for users of LLM agents by reducing inference costs and freeing up context windows, though it is incremental as it builds on existing prompt engineering techniques.
The paper tackles the problem of long system prompts in LLMs causing high latency and computational cost by proposing a method to replace them with a single token, achieving up to 3000x length reduction while retaining about 98% of downstream performance.
Carefully engineered system prompts play a critical role in guiding the behavior of LLM agents, but their considerable length introduces significant drawbacks, including increased inference latency, higher computational cost, and reduced effective context length. This raises the question of whether such lengthy prompts can be replaced by a drastically reduced number of tokens while preserving their behavioral effect on downstream tasks. To enable this, we propose a lightweight three-stage training framework that learns a single prompt-specific Behavior-Equivalent token ([BE]). The framework first trains [BE] to encode the natural-language content of the original system prompt via reconstruction, and then distills the prompt 's downstream behavior into this single token. Importantly, our method requires no access to model internals, no auxiliary compression models, and no labeled responses. Empirical evaluations on three datasets show that a single [BE] token achieves up to a 3000x reduction in prompt length, while retaining about 98% of the downstream performance of the original system prompts. This substantially reduces inference cost and leaves almost the entire context window available for user inputs.