When Does Context Help? Error Dynamics of Contextual Information in Large Language Models
This work addresses the theoretical gap in understanding contextual information's role in LLMs, which is incremental as it builds on existing settings like in-context learning.
The authors tackled the problem of understanding how contextual information influences large language models by developing a unified theoretical framework that characterizes contextual influence through output error dynamics, showing that error reduction requires specific geometric conditions and deriving an explicit upper bound for contextual correction. Their experiments validated the theory and led to a principled context selection strategy that improved performance by 0.6%.
Contextual information at inference time, such as demonstrations, retrieved knowledge, or interaction history, can substantially improve large language models (LLMs) without parameter updates, yet its theoretical role remains poorly understood beyond specific settings such as in-context learning (ICL). We present a unified theoretical framework for analyzing the effect of arbitrary contextual information in Transformer-based LLMs. Our analysis characterizes contextual influence through output error dynamics. In a single-layer Transformer, we prove that the context-conditioned error vector decomposes additively into the baseline error vector and a contextual correction vector. This yields necessary geometric conditions for error reduction: the contextual correction must align with the negative baseline error and satisfy a norm constraint. We further show that the contextual correction norm admits an explicit upper bound determined by context-query relevance and complementarity. These results extend to multi-context and multi-layer Transformers. Experiments across ICL, retrieval-augmented generation, and memory evolution validate our theory and motivate a principled context selection strategy that improves performance by $0.6\%$.