Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression
This addresses efficiency and performance issues in LLM applications, but it is incremental as it builds on existing retrieval-augmented methods.
The paper tackles the problem of irrelevant context in retrieval-augmented LLMs, which causes poor responses, latency, and costs, by introducing Instruction-Aware Contextual Compression to filter content. The result is a 50% reduction in context costs, 5% lower memory usage, 2.2x faster inference, with only a minor 0.047 Rouge-1 drop.
Large Language Models (LLMs) have garnered widespread attention due to their remarkable performance across various tasks. However, to mitigate the issue of hallucinations, LLMs often incorporate retrieval-augmented pipeline to provide them with rich external knowledge and context. Nevertheless, challenges stem from inaccurate and coarse-grained context retrieved from the retriever. Supplying irrelevant context to the LLMs can result in poorer responses, increased inference latency, and higher costs. This paper introduces a method called Instruction-Aware Contextual Compression, which filters out less informative content, thereby accelerating and enhancing the use of LLMs. The experimental results demonstrate that Instruction-Aware Contextual Compression notably reduces memory consumption and minimizes generation latency while maintaining performance levels comparable to those achieved with the use of the full context. Specifically, we achieved a 50% reduction in context-related costs, resulting in a 5% reduction in inference memory usage and a 2.2-fold increase in inference speed, with only a minor drop of 0.047 in Rouge-1. These findings suggest that our method strikes an effective balance between efficiency and performance.