CacheFocus: Dynamic Cache Re-Positioning for Efficient Retrieval-Augmented Generation
This addresses efficiency and scalability issues for users of long-context LLMs, representing an incremental improvement over existing caching and positional encoding techniques.
The paper tackles the problem of limited input lengths and high computational costs in Large Language Models by introducing CacheFocus, a method that enhances length normalization and reduces inference latency without additional training. Experiments show it outperforms alternative methods on Natural Questions and TriviaQA datasets even when inputs exceed the 4K limit of LLaMA-2, maintaining consistent performance with increasing documents.
Large Language Models (LLMs) excel across a variety of language tasks yet are constrained by limited input lengths and high computational costs. Existing approaches\textemdash such as relative positional encodings (e.g., RoPE, ALiBi) and sliding window mechanisms\textemdash partially alleviate these issues but often require additional training or suffer from performance degradation with longer inputs. In this paper, we introduce \textbf{\textit{CacheFocus}}, a method that enhances length normalization and reduces inference latency without any further training. Our approach leverages query-independent, offline caching to efficiently reuse a Context KV Cache Store. We address the amplification of abnormal token distributions problem by re-positioning cached keys and introducing Layer-Adaptive Cache Pruning to discard low-relevance caches during pre-filling. Additionally, our Adaptive Positional Allocation Strategy dynamically reassigns cache positions to maximize the use of the available positional encoding range. Experiments on the Natural Questions and TriviaQA datasets demonstrate that CacheFocus outperforms alternative methods even when inputs exceed the $4$K limit of the \texttt{LLaMA-2} model, emphasizing its practical effectiveness for long-context LLMs. Moreover, even with large maximum input length of \texttt{Qwen2}, the performance of CacheFocus shows that it maintains consistent performance even as the number of documents increases, effectively managing long-text generation without degradation.