DYCP: Dynamic Context Pruning for Long-Form Dialogue with LLMs
This addresses inefficiencies in context management for long-form dialogues with LLMs, offering a domain-specific improvement.
The paper tackles the problem of increased response latency and degraded answer quality in long-form dialogues with LLMs by introducing DyCP, a lightweight context management method that dynamically segments and retrieves relevant memory at query time, which consistently improves answer quality and reduces response latency across three benchmarks and multiple LLMs.
Large Language Models (LLMs) often exhibit increased response latency and degraded answer quality as dialogue length grows, making effective context management essential. However, existing methods rely on extra LLM calls to build memory or perform offline memory construction without considering the current user utterance, which can introduce inefficiencies or disrupt conversational continuity. We introduce DyCP, a lightweight context management method that dynamically segment and retrieve relevant memory at query time. It preserves the sequential structure of dialogue without predefined topic boundaries and supports efficient, adaptive context retrieval. Across three long-form dialogue benchmarks, LoCoMo, MT-Bench+, and SCM4LLMs, and multiple LLMs, DyCP consistently improves answer quality while reducing response latency. We also examine the gap between modern LLMs' expanded context windows and their actual long-context processing capacity, highlighting the continued importance of effective context management.