SideQuest: Model-Driven KV Cache Management for Long-Horizon Agentic Reasoning
This work tackles the critical problem of efficient memory management for Large Language Models during long-horizon agentic reasoning, which is crucial for researchers and agents performing multi-hop reasoning over extensive information.
This paper addresses the rapid growth of KV cache memory usage in long-running agentic tasks by proposing SideQuest, a novel approach where the Large Reasoning Model (LRM) itself manages KV cache compression. SideQuest reduces peak token usage by up to 65% on agentic tasks with minimal accuracy degradation, outperforming heuristic-based methods.
Long-running agentic tasks, such as deep research, require multi-hop reasoning over information distributed across multiple webpages and documents. In such tasks, the LLM context is dominated by tokens from external retrieval, causing memory usage to grow rapidly and limiting decode performance. While several KV cache compression techniques exist for long-context inputs, we find that existing heuristics fail to support multi-step reasoning models effectively. We address this challenge with SideQuest -- a novel approach that leverages the Large Reasoning Model (LRM) itself to perform KV cache compression by reasoning about the usefulness of tokens in its context. To prevent the tokens associated with this management process from polluting the model's memory, we frame KV cache compression as an auxiliary task executed in parallel to the main reasoning task. Our evaluations, using a model trained with just 215 samples, show that SideQuest reduces peak token usage by up to 65% on agentic tasks with minimal degradation in accuracy, outperforming heuristic-based KV cache compression techniques.