Prompt Compression via Activation Aggregation
This work addresses the problem of reducing computational cost for repeated instruction prompts in LLMs, offering a practical compression technique with minimal accuracy loss.
The authors propose compressing instruction prompts into a single activation vector via a learned weighted sum of intermediate-layer activations, which is then injected into an early layer of the LLM. This method achieves under 2% accuracy drop relative to full prompt processing, enabling per-query computation reduction for fixed prompts.
Large language models process prompts by propagating activations through dozens of layers before generating a response. We ask whether the task-relevant information contained in an instruction prompt can be compressed into a single activation vector and re-injected into the model, replacing the original token sequence? We show this is achievable using a learned weighted sum of activations extracted at an intermediate layer and injected at an early layer of the target LLM. The compressed vector preserves task-relevant information, incurring an accuracy drop of under $2\%$ relative to full prompt processing. Beyond its practical implications, including reducing per-query computation for fixed instruction prompts without reprocessing the original token sequence, our analysis reveals structure in the activation space of LLMs: (i) mid-layer representations transfer meaningfully to early layers, suggesting a degree of cross-layer compatibility in how information is encoded; (ii) a single activation vector encodes a quantifiable and recoverable amount of semantic information; (iii) a weighted sum of activations is a robust representation compressor.