Directed Information $γ$-covering: An Information-Theoretic Framework for Context Engineering
This addresses context selection inefficiencies in LLM pipelines, offering a query-agnostic solution that reduces online costs, though it is incremental as it builds on existing information-theoretic concepts.
The paper tackles the problem of redundancy-aware context engineering for LLMs by introducing Directed Information γ-covering, a framework that selects context chunks based on causal predictiveness to preserve query information with bounded slack. Experiments on HotpotQA show it consistently improves over BM25, with advantages in context compression and single-slot prompt selection.
We introduce \textbf{Directed Information $γ$-covering}, a simple but general framework for redundancy-aware context engineering. Directed information (DI), a causal analogue of mutual information, measures asymmetric predictiveness between chunks. If $\operatorname{DI}_{i \to j} \ge H(C_j) - γ$, then $C_i$ suffices to represent $C_j$ up to $γ$ bits. Building on this criterion, we formulate context selection as a $γ$-cover problem and propose a greedy algorithm with provable guarantees: it preserves query information within bounded slack, inherits $(1+\ln n)$ and $(1-1/e)$ approximations from submodular set cover, and enforces a diversity margin. Importantly, building the $γ$-cover is \emph{query-agnostic}: it incurs no online cost and can be computed once offline and amortized across all queries. Experiments on HotpotQA show that $γ$-covering consistently improves over BM25, a competitive baseline, and provides clear advantages in hard-decision regimes such as context compression and single-slot prompt selection. These results establish DI $γ$-covering as a principled, self-organizing backbone for modern LLM pipelines.