Local Causal Attribution of Chain-of-Thought Reasoning
For researchers and practitioners needing to understand and verify the reasoning process of language models, this work provides a more faithful attribution method for chain-of-thought traces.
The authors propose AttriCoT, a black-box method for local causal attribution of chain-of-thought reasoning in language models, which estimates importance parameters in a structural causal model using O(U) forward passes. Evaluated on 5 datasets and 4 reasoning models, AttriCoT produces more faithful attributions than alternatives and reveals differences in thought structure across models and domains.
Understanding the causal structure of a language model's thought process is a problem of significant importance for both transparency and safety. In this work, we take a local approach toward this goal by analyzing the causal relationships among individual components, termed units, of a given, specific chain-of-thought trace. We construct a structural causal model on these units and relate each unit to the log probability of generating (subsequent) output units. Our algorithm, termed AttriCoT, is a black-box method that performs attribution by estimating importance parameters in the structural causal model using $O(U)$ forward passes through the model, where $U$ is the number of units. Evaluation of perturbation curves across 5 datasets and 4 reasoning models shows that AttriCoT produces attributions that are more faithful to the model's behavior than alternative methods. The attribution results also reveal notable differences in thought structure between models and domains.