Position IDs Matter: An Enhanced Position Layout for Efficient Context Compression in Large Language Models
This addresses the issue of context compression inefficiency for users of large language models, but it is incremental as it builds on existing compression methods by adjusting position IDs.
The paper tackled the problem of inefficient context compression in large language models due to local inductive biases from position encodings, and the result was a 1.9 ROUGE-1 F1 improvement on out-of-domain question answering datasets and a 2.6-point accuracy gain for vision compression LLMs with their Enhanced Position Layout method.
Using special tokens (e.g., gist, memory, or compressed tokens) to compress context information is a common practice for large language models (LLMs). However, existing approaches often neglect that position encodings inherently induce local inductive biases in models, causing the compression process to ignore holistic contextual dependencies. We propose \textbf{Enhanced Position Layout (EPL)}, a simple yet effective method that improves the context compression capability of LLMs by only adjusting position IDs, the numerical identifiers that specify token positions. EPL minimizes the distance between context tokens and their corresponding special tokens and at the same time maintains the sequence order in position IDs between context tokens, special tokens, and the subsequent tokens. Integrating EPL into our best performing context compression model results in a 1.9 ROUGE-1 F1 improvement on out-of-domain question answering datasets on average. When extended to multimodal scenarios, EPL leads to an average accuracy gain of 2.6 points for vision compression LLMs.