ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models
This addresses the issue of unfaithful outputs in LLMs for applications relying on up-to-date or external knowledge, though it is incremental as it builds on existing activation steering methods.
The paper tackles the problem of large language models (LLMs) defaulting to memorized facts instead of faithfully following externally retrieved context when conflicts arise, introducing ContextFocus, a lightweight activation steering approach that significantly improves contextual faithfulness without finetuning or significant inference overhead.
Large Language Models (LLMs) encode vast amounts of parametric knowledge during pre-training. As world knowledge evolves, effective deployment increasingly depends on their ability to faithfully follow externally retrieved context. When such evidence conflicts with the model's internal knowledge, LLMs often default to memorized facts, producing unfaithful outputs. In this work, we introduce ContextFocus, a lightweight activation steering approach that improves context faithfulness in such knowledge-conflict settings while preserving fluency and efficiency. Unlike prior approaches, our solution requires no model finetuning and incurs minimal inference-time overhead, making it highly efficient. We evaluate ContextFocus on the ConFiQA benchmark, comparing it against strong baselines including ContextDPO, COIECD, and prompting-based methods. Furthermore, we show that our method is complementary to prompting strategies and remains effective on larger models. Extensive experiments show that ContextFocus significantly improves contextual-faithfulness. Our results highlight the effectiveness, robustness, and efficiency of ContextFocus in improving contextual-faithfulness of LLM outputs.