Entropic Context Shaping: Information-Theoretic Filtering for Context-Aware LLM Agents
This addresses context selection for LLM agents, showing pragmatic utility outperforms lexical similarity, though it appears incremental as it builds on existing information-theoretic concepts for a specific bottleneck.
The paper tackles the problem of distinguishing useful from misleading information in context engineering for LLM agents by introducing Entropic Context Shaping (ECS), an information-theoretic framework that measures context utility via shifts in answer distributions toward correct answers. On turn-level context selection, ECS with Llama-3.1-8B achieves F1=0.265, a 71.83% relative improvement over TF-IDF (F1=0.154).
Context engineering for large language model (LLM) agents requires distinguishing pragmatically useful information from misleading distractors. We introduce Entropic Context Shaping (ECS), an information-theoretic framework that measures context utility via the shift in the model's answer distribution toward the correct answer. Unlike lexical similarity methods that rely on word overlap, ECS captures pragmatic utility -- whether a passage actually helps answer the question. We formalize utility as the signed change in answer probability and provide theoretical analysis showing that task-irrelevant updates yield near-zero distribution shift. We evaluate on multi-turn context selection tasks using LongMemEval (session-level) and LoCoMo (turn-level) benchmarks. On fine-grained turn selection, ECS with Llama-3.1-8B achieves F1=0.265, a 71.83% relative improvement over TF-IDF (F1=0.154), demonstrating that pragmatic utility outperforms lexical similarity when precise context selection matters. Code and data are available in the supplementary materials.