Long Context In-Context Compression by Getting to the Gist of Gisting
This addresses a critical bottleneck for practical LLM adoption by improving in-context compression without architectural changes, though it is incremental as it builds on prior gisting methods.
The paper tackled the problem of long context compression for LLMs, showing that existing gisting methods struggle with performance drops on longer contexts, and proposed GistPool, which significantly boosts performance while maintaining simplicity.
Long context processing is critical for the adoption of LLMs, but existing methods often introduce architectural complexity that hinders their practical adoption. Gisting, an in-context compression method with no architectural modification to the decoder transformer, is a promising approach due to its simplicity and compatibility with existing frameworks. While effective for short instructions, we demonstrate that gisting struggles with longer contexts, with significant performance drops even at minimal compression rates. Surprisingly, a simple average pooling baseline consistently outperforms gisting. We analyze the limitations of gisting, including information flow interruptions, capacity limitations and the inability to restrict its attention to subsets of the context. Motivated by theoretical insights into the performance gap between gisting and average pooling, and supported by extensive experimentation, we propose GistPool, a new in-context compression method. GistPool preserves the simplicity of gisting, while significantly boosting its performance on long context compression tasks.