Task Matters: Knowledge Requirements Shape LLM Responses to Context-Memory Conflict
For LLM developers and evaluators, reveals that context-memory conflict handling is not uniform across tasks, motivating task-aware deployment and evaluation strategies.
LLMs face context-memory conflicts, but prior work assumed tasks should always rely on context. This study shows conflict effects are task-dependent: performance degrades based on knowledge demands and conflict plausibility, and strategies like rationales can harm tasks needing parametric knowledge, biasing LLM-as-judge evaluations.
Large language models (LLMs) draw on both contextual information and parametric memory, yet these sources can conflict. Prior studies have largely examined this issue in contextual question answering, implicitly assuming that tasks should rely on the provided context, leaving unclear how LLMs behave when tasks require different types and degrees of knowledge utilization. We address this gap with a model-agnostic diagnostic framework that holds underlying knowledge constant while introducing controlled conflicts across tasks with varying knowledge demands. Experiments on representative open-weight and proprietary LLMs show that performance degradation under conflict is driven by both task-specific knowledge reliance and conflict plausibility; that strategies such as rationales or context reiteration increase context reliance, helping context-only tasks but harming those requiring parametric knowledge; and that these effects bias model-based evaluation, calling into question the reliability of LLMs as judges. Overall, our findings reveal that context-memory conflict is inherently task-dependent and motivate task-aware approaches to balancing context and memory in LLM deployment and evaluation.