Assessing LLM Reasoning Through Implicit Causal Chain Discovery in Climate Discourse
This work addresses the problem of assessing LLM reasoning capabilities for implicit causal chains in argumentation, particularly in polarized climate discussions, though it is incremental as it builds on existing diagnostic frameworks.
The study evaluated the mechanistic causal reasoning of nine large language models (LLMs) by having them generate intermediate causal steps for cause-effect pairs from climate discourse, finding that while the models produced logically coherent chains, their judgments relied more on associative pattern matching than genuine causal reasoning.
How does a cause lead to an effect, and which intermediate causal steps explain their connection? This work scrutinizes the mechanistic causal reasoning capabilities of large language models (LLMs) to answer these questions through the task of implicit causal chain discovery. In a diagnostic evaluation framework, we instruct nine LLMs to generate all possible intermediate causal steps linking given cause-effect pairs in causal chain structures. These pairs are drawn from recent resources in argumentation studies featuring polarized discussion on climate change. Our analysis reveals that LLMs vary in the number and granularity of causal steps they produce. Although they are generally self-consistent and confident about the intermediate causal connections in the generated chains, their judgments are mainly driven by associative pattern matching rather than genuine causal reasoning. Nonetheless, human evaluations confirmed the logical coherence and integrity of the generated chains. Our baseline causal chain discovery approach, insights from our diagnostic evaluation, and benchmark dataset with causal chains lay a solid foundation for advancing future work in implicit, mechanistic causal reasoning in argumentation settings.