CausalBench: A Comprehensive Benchmark for Causal Learning Capability of LLMs
This work addresses the problem of evaluating LLMs' causal reasoning for researchers and developers, providing a benchmark that reveals limitations and strengths, though it is incremental in benchmarking rather than proposing new methods.
The paper tackles the lack of a comprehensive benchmark for evaluating the causal learning capabilities of large language models (LLMs) by developing CausalBench, which includes three tasks of varying difficulty. Evaluations of 19 leading LLMs show that closed-source models lag behind traditional algorithms on larger-scale networks (>50 nodes), struggling with collider structures but excelling at chain structures.
The ability to understand causality significantly impacts the competence of large language models (LLMs) in output explanation and counterfactual reasoning, as causality reveals the underlying data distribution. However, the lack of a comprehensive benchmark currently limits the evaluation of LLMs' causal learning capabilities. To fill this gap, this paper develops CausalBench based on data from the causal research community, enabling comparative evaluations of LLMs against traditional causal learning algorithms. To provide a comprehensive investigation, we offer three tasks of varying difficulties, including correlation, causal skeleton, and causality identification. Evaluations of 19 leading LLMs reveal that, while closed-source LLMs show potential for simple causal relationships, they significantly lag behind traditional algorithms on larger-scale networks ($>50$ nodes). Specifically, LLMs struggle with collider structures but excel at chain structures, especially at long-chain causality analogous to Chains-of-Thought techniques. This supports the current prompt approaches while suggesting directions to enhance LLMs' causal reasoning capability. Furthermore, CausalBench incorporates background knowledge and training data into prompts to thoroughly unlock LLMs' text-comprehension ability during evaluation, whose findings indicate that, LLM understand causality through semantic associations with distinct entities, rather than directly from contextual information or numerical distributions.