A Critical Review of Causal Reasoning Benchmarks for Large Language Models
This work addresses the problem of accurately evaluating causal reasoning in LLMs for researchers and developers, but it is incremental as it reviews and synthesizes existing benchmarks rather than introducing new methods.
The paper reviews existing benchmarks for evaluating causal reasoning in Large Language Models, highlighting that many can be solved via knowledge retrieval rather than true causal inference, and proposes criteria for more effective benchmarks to assess causal understanding.
Numerous benchmarks aim to evaluate the capabilities of Large Language Models (LLMs) for causal inference and reasoning. However, many of them can likely be solved through the retrieval of domain knowledge, questioning whether they achieve their purpose. In this review, we present a comprehensive overview of LLM benchmarks for causality. We highlight how recent benchmarks move towards a more thorough definition of causal reasoning by incorporating interventional or counterfactual reasoning. We derive a set of criteria that a useful benchmark or set of benchmarks should aim to satisfy. We hope this work will pave the way towards a general framework for the assessment of causal understanding in LLMs and the design of novel benchmarks.