Can LLMs Reconcile Knowledge Conflicts in Counterfactual Reasoning
This work highlights limitations in LLMs' adaptability for novel settings, which is crucial for real-world applications requiring flexible reasoning.
The study investigated whether large language models can integrate new information with their stored knowledge for counterfactual reasoning, finding that they generally fail and rely on parametric knowledge, with finetuning often degrading stored knowledge.
Large Language Models have been shown to contain extensive world knowledge in their parameters, enabling impressive performance on many knowledge intensive tasks. However, when deployed in novel settings, LLMs often encounter situations where they must integrate parametric knowledge with new or unfamiliar information. In this work, we explore whether LLMs can combine knowledge in-context with their parametric knowledge through the lens of counterfactual reasoning. Through synthetic and real experiments in multi-hop reasoning problems, we show that LLMs generally struggle with counterfactual reasoning, often resorting to exclusively using their parametric knowledge. Moreover, we show that simple post-hoc finetuning can struggle to instill counterfactual reasoning ability -- often leading to degradation in stored parametric knowledge. Ultimately, our work reveals important limitations of current LLM's abilities to re-purpose parametric knowledge in novel settings.