Counterfactual reasoning: an analysis of in-context emergence
This work addresses the problem of understanding and enhancing counterfactual reasoning in AI models, which is incremental but provides mechanistic insights for researchers in interpretability and reasoning.
This paper investigates whether language models can perform counterfactual reasoning by predicting outcomes of hypothetical scenarios, using a synthetic linear regression task that requires inferring latent concepts and copying contextual noise. The authors demonstrate that Transformers are capable of this reasoning, with performance driven by self-attention, model depth, and pre-training data diversity, and they identify specific neural mechanisms like noise abduction heads.
Large-scale neural language models exhibit remarkable performance in in-context learning: the ability to learn and reason about the input context on the fly. This work studies in-context counterfactual reasoning in language models, that is, the ability to predict consequences of a hypothetical scenario. We focus on a well-defined, synthetic linear regression task that requires noise abduction. Accurate prediction is based on (1) inferring an unobserved latent concept and (2) copying contextual noise from factual observations. We show that language models are capable of counterfactual reasoning. Further, we enhance existing identifiability results and reduce counterfactual reasoning for a broad class of functions to a transformation on in-context observations. In Transformers, we find that self-attention, model depth and pre-training data diversity drive performance. Moreover, we provide mechanistic evidence that the latent concept is linearly represented in the residual stream and we introduce designated \textit{noise abduction heads} central to performing counterfactual reasoning. Lastly, our findings extend to counterfactual reasoning under SDE dynamics and reflect that Transformers can perform noise abduction on sequential data, providing preliminary evidence on the potential for counterfactual story generation. Our code is available under https://github.com/mrtzmllr/iccr.