Decoding Answers Before Chain-of-Thought: Evidence from Pre-CoT Probes and Activation Steering
This work addresses interpretability concerns for researchers and practitioners using CoT in LLMs, showing it can be unfaithful and lead to undesirable behaviors, which is incremental as it builds on existing CoT research.
The study tackled the problem of assessing the faithfulness of chain-of-thought (CoT) reasoning in large language models by investigating whether models determine answers before generating CoT, finding that linear probes on activations could predict final answers with 0.9 AUC and steering activations flipped answers in over 50% of cases, revealing failure modes like non-entailment and confabulation.
As chain-of-thought (CoT) has become central to scaling reasoning capabilities in large language models (LLMs), it has also emerged as a promising tool for interpretability, suggesting the opportunity to understand model decisions through verbalized reasoning. However, the utility of CoT toward interpretability depends upon its faithfulness -- whether the model's stated reasoning reflects the underlying decision process. We provide mechanistic evidence that instruction-tuned models often determine their answer before generating CoT. Training linear probes on residual stream activations at the last token before CoT, we can predict the model's final answer with 0.9 AUC on most tasks. We find that these directions are not only predictive, but also causal: steering activations along the probe direction flips model answers in over 50% of cases, significantly exceeding orthogonal baselines. When steering induces incorrect answers, we observe two distinct failure modes: non-entailment (stating correct premises but drawing unsupported conclusions) and confabulation (fabricating false premises). While post-hoc reasoning may be instrumentally useful when the model has a correct pre-CoT belief, these failure modes suggest it can result in undesirable behaviors when reasoning from a false belief.